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CLEANROOM  PAMPHLET 


“Software  products  with  reliable  operating  characteristics  have  been  notoriously  difficult 
to  develop.  The  Cleanroom  method  introduces  sound  software  engineering  into  the  development 
cycle  and  provides  the  quality  control  that’s  essential  for  product  success.” 1 

This  pamphlet  is  part  of  the  ongoing  effort  by  the  Software  Technology  Support  Center 
(STSC)  in  assisting  Air  Force  organizations  to  identify,  evaluate  and  adopt  technologies  that  will 
improve:  (1)  the  quality  of  their  products,  (2)  their  efficiency  in  producing  those  products,  and  (3) 
their  ability  to  accurately  predict  the  cost  and  schedule  of  their  delivery.  The  publication  of  this 
pamphlet  makes  no  implications  that  the  STSC  is  endorsing  the  Cleanroom  process  over  any 
other  technology.  The  STSC  just  feels  that  this  is  a  technology  that  is  worth  looking  into  as  a 
method  for  improving  software  quality,  and  possibly  increasing  productivity.  The  Cleanroom 
process  may  be  a  viable  method  for  some  organizations  and  may  not  work  for  others.  Cleanroom, 
just  like  any  other  technology,  is  something  that  needs  to  be  evaluated  by  the  individual 
organizations. 

The  STSC’s  Software  Quality  Engineering  (SQE)  Team  is  responsible  for  providing 
products  and  services  pertaining  to  software  quality.  Members  of  the  SQE  Team  are:  Bryce 
Ragland,  Johnnie  Henderson  (LORAL),  and  Mark  Dawood  (SAIC). 

The  following  items  are  included  in  the  Cleanroom  Pamphlet: 

•  “Cleanroom  Software  Engineering:  Management  Overview”,  Richard  C.  Linger 

•  “Adopting  Cleanroom  software  engineering  with  a  phased  approach”,  P.  A.  Hausler, 

R.  C.  Linger,  C.  J.  Trammell,  IBM  Systems  Journal  vol.  33,  no.  1, 1994 

•  “Cleanroom  Process  Model”,  Richard  C.  Linger,  IEEE  Software,  March  1994 

•  “Experience  Using  Cleanroom  Software  Engineering  in  the  US  Army”,  S.  Wayne 
Sherer,  Paul  G.  Arnold,  Ara  Kouchakdjian,  Proceeding  from  STC’94  (updated) 

•  “Why  Isn’t  Cleanroom  the  Universal  Software  Development  Methodology?”,  Johnnie 
Henderson 

•  Bibliography  of  Cleanroom  Articles/Books 

•  Listing  of  Organizations  that  assist  with  Cleanroom  Adoption 


1  Michael  Dyer,  The  Cleanroom  Approach  to  Quality  Software  Development,  John  Wiley  &  Sons,  Inc.  New  York, 
1994  (Back  Cover) 
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Developing  Software  Under  Statistical  Quality  Control 

Cleanroom  software  engineering  is  a  theory-based,  team-oriented  process  for  on-schedule 
development  and  certification  of  ultra-high-reliability  software  systems  with  improved  productivity 
under  statistical  quality  control  [2,3].  The  Cleanroom  name  is  borrowed  from  hardware 
Cleanrooms,  with  their  emphasis  on  prevention  of  errors  through  engineering  discipline,  rather  than 
error  removal.  Cleanroom  combines  rigorous  methods  of  software  specification,  design, 
correctness  verification,  and  statistical  quality  certification  in  a  new  life  cycle  model  based  on 
incremental  development. 

You  can  use  the  Cleanroom  process  and  supporting  technologies  to  develop  software  systems  that 
approach  zero  defects  and  have  scientifically  certified  reliability  for  operational  field  use. 

In  contrast  to  traditional  development  approaches,  in  the  Cleanroom  process  you  embed  software 
development  and  testing  within  a  formal  statistical  quality  control  process.  In  such  a  process, 
software  engineering  is  required  to  create  software  that  approaches  zero  defects  and  can  enter 
system  testing  directly  [1].  Then  statistical  usage-based  testing  is  used  to  provide  statistical 
inferences  about  the  reliability  of  the  software.  This  systematic  process  of  assessing  and  controlling 
software  quality  during  development  permits  you  to  certify  product  reliability  at  delivery,  based  on 
a  complete  public  record  of  the  testing  and  all  engineering  change  activity  required  to  deliver 
acceptable  software. 

The  significance  of  a  process  under  statistical  quality  control  is  well  illustrated  by  modem 
manufacturing  techniques  where  the  sampling  of  output  is  directly  fed  back  into  the  process  to 
control  quality.  Once  the  discipline  of  statistical  quality  control  is  in  place,  management  has 
objective  visibility  into  the  software  development  process  and  can  control  process  changes  to 
control  product  quality. 

Key  ingredients  of  the  Cleanroom  process  are  a  new  development  life  cycle  and  independent 
quality  assessment  through  statistical  testing.  You  begin  the  development  life  cycle  with  a 
specification  that  not  only  defines  functional  requirements,  but  also  identifies  statistical  usage  of  the 
software  and  a  nested  sequence  of  user-function  subsets  that  can  be  released  and  tested  as 
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increments  which  accumulate  into  the  final  system.  Rigorous  software  engineering  methods 
provide  design  and  correctness  verification  techniques  to  create  provably  correct  software. 
Correctness  verification  by  software  engineering  teams  has  proven  to  be  a  powerful  and  effective 
process  for  approaching  zero  defects  prior  to  any  execution  of  the  software. 

You  place  software  under  engineering  change  control  from  first  execution  on,  and  all  execution  is 
controlled  by  an  independent  certification  team  that  uses  statistical  testing  methods  to  evaluate 
software  quality.  Traditional  structural  or  coverage  tests,  no  matter  how  carefully  selected  or  how 
comprehensive  their  test  plans,  provide  only  elaborate  anecdotes  of  quality,  with  no  basis  for 
scientific  statistical  extrapolation  to  operational  environments.  However,  statistical  testing  in  a 
quality  control  process  results  in  objective  quality  certification  of  software  at  acceptance. 


Cleanroom  Results 

The  Cleanroom  process  has  been  developed,  reduced  to  practice,  and  demonstrated  in  development 
of  a  variety  of  software  systems  in  a  various  languages  and  environments.  Published  results  of 
projects  totaling  over  a  million  lines  of  code  earned  out  by  IBM,  STARS,  NASA,  and  other 
organizations  [1,2]  have  shown  substantial  improvements  over  traditional  results,  as  the  following 
examples  illustrate: 

Quality.  Improvements  of  10-20X  and  more  over  baseline  performance  have  been  achieved. 
Some  Cleanroom-developed  systems  have  experienced  no  errors  whatsoever  in  field  use.  For 
example,  IBM  developed  an  embedded,  real-time,  bus  architecture,  multiple-processor  device 
controller  product  that  experienced  no  errors  in  two  years  use  at  over  300  customer  locations. 

Productivity.  Gains  of  1.5-5X  over  baseline  performance  have  been  reported.  For  example,  an 
Ericsson  Telecom  project  to  develop  a  374  KLOC  operating  system  reported  gains  of  1.7X  in 
development  productivity,  1.6X  in  testing  productivity,  and  an  IBM  project  to  develop  a  network 
management  and  outage  avoidance  product  reported  2X  improvement  in  development  productivity. 

Life  cycle  costs.  Dramatic  reductions  have  been  achieved  due  to  sharp  decreases  in  error  correction 
and  maintenance  costs  over  the  life  of  a  product.  For  example,  IBM  developed  a  COBOL 
structuring  product  that  experienced  just  seven  minor  errors  in  the  first  three  years  of  field  use,  all 
simple  fixes,  and  required  a  small  fraction  of  the  maintenance  budget  associated  with  products  of 
similar  complexity. 

Return  on  investment.  Experience  shows  that  Cleanroom  adoption  costs  can  be  recovered  on  the 
first  project.  For  example,  an  11  to  1  ROI  was  reported  by  the  Picatinny  Arsenal  STARS 
Cleanroom  project,  with  the  investment  covering  all  Cleanroom  training  and  consultation  costs. 
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You  can  experience  other  benefits  of  the  Cleanroom  process  that  are  more  difficult  to  quantify,  but 
are  real  nonetheless.  For  example,  Cleanroom  statistical  testing  provides  you  with  scientific 
measures  of  product  quality  for  the  first  time,  permitting  objective  decision-making  on  whether  and 
when  to  stop  testing  and  release  products.  It  also  provides  scientific  projections  of  quality  in  field 
use,  the  only  known  method  for  doing  this.  In  addition,  substantial  increases  in  the  job  satisfaction 
of  Cleanroom  teams  have  been  reported  [8]. 

You  can  apply  the  Cleanroom  process  to  development  of  new  systems  and  maintenance,  evolution, 
and  re-engineering  of  existing  systems.  It  is  language,  environment,  and  subject-matter 
independent,  and  can  be  used  to  develop  and  evolve  a  variety  of  systems,  including  real-time, 
embedded,  host,  distributed,  workstation,  client-server,  and  microcode  systems.  Cleanroom  is 
compatible  with  prototyping  and  promotes  reuse  through  precise  definition  of  component 
functional  semantics  and  certification  of  component  reliability. 

Cleanroom  is  compatible  with  and  supports  performance  at  SEI  Capability  Maturity  Model  levels  2 
through  5  [7].  Organizations  at  CMM  level  1  may  wish  to  achieve  level  2  before  embarking  on 
Cleanroom  operations. 


Cleanroom  Technologies 

The  Cleanroom  process  incorporates  technologies  for  management,  development,  and  testing,  as 
follows: 

Incremental  Development.  Cleanroom  management  is  based  on  incremental  development  and 
certification  of  a  pipeline  of  user-function  increments  that  accumulate  into  the  final  product. 
System  integration  is  top-down  and  continual,  with  system  functionality  growing  through  addition 
of  successive  increments.  Incremental  development  enables  early  and  continual  assessment  of 
product  quality  and  user  feedback,  and  facilitates  improvements  as  development  progresses.  The 
incremental  approach  permits  controlled,  stepwise  integration  of  components,  avoiding  the  risky, 
last-minute  integration  often  experienced  in  traditional  development. 

Rigorous  Specification  and  Design.  The  development  team  produces  software  approaching  zero 
defects  through  use  of  a  rigorous  stepwise  refinement  and  verification  process  for  specification  and 
design  using  object-based  Box  Structure  technology  [6].  Box  Structures  permit  precise  definition 
of  required  user  function  and  system  object  architecture,  and  scale  up  to  maintain  intellectual 
control  in  large  system  development.  A  key  concept  in  Box  Structures  is  referential  transparency, 
whereby  the  subspecifications  for  successive  object  refinements  are  developed,  connected  and 
verified  in  a  coherent  structure  prior  to  their  independent  elaboration.  In  Cleanroom,  correctness  is 
built  in,  not  tested  in. 
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Correctness  Verification.  All  Cleanroom-developed  software  is  subject  to  rigorous  correctness 
verification  by  the  development  team  prior  to  release  to  the  certification  test  team.  A  practical  and 
powerful  process,  verification  permits  development  teams  to  completely  verify  the  correctness  of 
software  with  respect  to  specifications.  A  Correctness  Theorem  defines  conditions  to  be  met  for 
achieving  zero-defect  software  [4].  These  conditions  are  verified  in  mental/verbal  proofs  of 
correctness  in  development  team  reviews.  Even  though  programs  of  any  size  contain  a  virtually 
infinite  number  of  paths,  the  theorem  reduces  verification  to  a  finite  number  of  checks  and  ensures 
that  all  software  logic  is  completely  verified  in  all  possible  circumstances  of  use.  The  verification 
step  is  extremely  powerful  in  eliminating  defects,  and  is  a  major  factor  in  the  dramatic  quality 
improvements  experienced  by  Cleanroom  teams. 

Statistical  Quality  Certification.  The  objective  of  the  certification  test  team  is  to  provide 
scientific  certification  of  software  reliability,  not  to  test  quality  in,  an  impossible  task.  Following 
correctness  verification,  software  increments  are  placed  under  engineering  change  control  and 
undergo  first  execution.  Statistical  usage  testing  is  earned  out  to  produce  scientifically  valid 
measures  of  software  quality  and  reliability  [5].  The  statistical  usage  approach  tests  software  the 
way  users  intend  to  use  it.  Test  cases  are  generated  based  on  usage  probability  distributions  that 
model  anticipated  software  use  in  all  possible  circumstances,  including  unusual  and  stress 
situations.  Usage  distributions  can  be  defined  in  Markov  models  that  permit  substantial 
management  analysis  and  simulation  of  test  operations,  as  well  as  automatic  test  case  generation 
[9].  Objective  statistical  measures  of  software  reliability,  such  as  Mean  Time  To  Failure  (MTTF), 
are  computed  based  on  test  results  for  informed  management  decision-making.  Because  statistical 
usage  testing  is  biased  toward  detection  of  more  serious,  high-frequency  errors  first,  it  is  more 
effective  at  improving  software  reliability  in  less  time  than  traditional  testing  techniques. 


Cleanroom  Application 

You  can  apply  Cleanroom  practices  in  the  following  environments: 

New  Systems.  You  can  use  the  Cleanroom  process  to  provide  a  coherent  management  and 
technical  framework  for  on-schedule  development  under  intellectual  control.  Incremental 
development  provides  for  early  quality  assessment  and  user  feedback  on  system  function,  and 
avoids  the  risk  associated  with  late  component  integration  in  waterfall-based  developments. 

Existing  Systems.  You  can  develop  modifications  and  extensions  to  existing  systems  using 
Cleanroom  technology.  In  addition,  problem-prone  modules  in  existing  systems  can  be  re¬ 
engineered  to  Cleanroom  quality  through  use  of  design  abstraction  and  correctness  verification 
techniques. 

Cleanroom  Acquisition.  You  can  integrate  the  Cleanroom  process  into  acquisition  practices  in 
terms  of  required  project  processes  and  deliverables.  For  example,  an  incremental  development 


process  can  be  required,  with  incremental  deliverables  associated  with  rigorous  software 
specification,  design,  correctness  verification,  and  statistical  reliability  certification.  Required 
software  MTTF  and  other  statistical  measures  can  be  specified  as  prerequisites  for  software 
acceptance  at  delivery. 


Phased  Introduction  of  Cleanroom 

You  can  introduce  the  Cleanroom  process  into  an  organization  in  a  staged  manner.  A  successful 
strategy  has  been  to  start  with  pilot  projects  and  teams  for  Cleanroom  development.  Success  with 
these  projects  provides  incentives  for  widespread  adoption.  As  use  of  the  Cleanroom  process 
grows,  experienced  members  of  early  teams  can  become  leaders  of  new  teams. 

Within  a  project,  a  phased  introduction  of  Cleanroom  process  and  technology  elements  is  possible 
[1].  As  experience  accumulates,  you  can  adopt  successive  elements  of  Cleanroom  in  a  staged 
implementation. 


References 

1.  Hausler,  P.  A.,  R.  C.  Linger,  and  C.  T.  Trammell,  "Adopting  Cleanroom  Software 
Engineering  with  a  Phased  Approach,"  IBM  Systems  Journal,  March,  1994. 

2.  Linger,  R.  C.,  "Cleanroom  Software  Engineering  for  Zero-Defect  Software,"  Proceedings  of 
15th  International  Conference  on  Software  Engineering,  IEEE  Computer  Society  Press,  Los 
Alamitos,  CA,  1993. 

3.  Linger,  R.  C.,  “Cleanroom  Process  Model,”  IEEE  Software \  IEEE  Computer  Society  Press, 
Los  Alamitos,  CA,  Vol.  1 1,  No.  2,  March,  1994. 

4.  Linger,  R.  C.,  and  H.  D.  Mills,  and  B.  I.  Witt,  Structured  Programming:  Theory  and 
Practice,  Addison-Wesley,  Reading,  MA,  1979. 

5.  Mills,  H.  D.,  “Certifying  the  Correctness  of  Software,”  Proceedings  of  25th  Hawaii 
International  Conference  on  System  Sciences,  IEEE  Computer  Society  Press,  January,  1992, 
pp.  373-381. 

6.  Mills,  H.  D.,  R.  C.  Linger,  and  A.  R.  Hevner,  Principles  of  Information  Systems  Analysis 
and  Design,  Academic  Press,  New  York,  1986. 


\ 

\ 


5 


7.  Paulk,  M.  C.,  et  al„  Capability  Maturity  Model  for  Software,  Version  1.1  (CMU/SEI-93- 
TR-25  ADA263432).  Pittsburgh,  PA,  Software  Engineering  Institute,  Carnegie  Mellon 
University,  1993. 

8.  Sherer,  S.  W„  P.  G.  Arnold,  and  A.  Kouchakdjian,  “Experience  Using  Cleanroom  Software 
Engineering  in  the  US  Army,”  Proceedings  of  Second  Annual  European  Industrial 
Symposium  on  Cleanroom  Software  Engineering,  Q-Labs  AB,  IDEON  Research  Park,  S- 
223  70  Lund,  Sweden. 

9.  Whittaker,  J.  A.  and  M.  G.  Thomason,  “A  Markov  Chain  Model  for  Statistical  Software 
Testing,”  IEEE  Transactions  on  Software  Engineering,  IEEE  Computer  society  Press,  Los 
Alamitos,  CA,  Vol.  20,  No.  4,  October,  1994. 


6 


Reprinted  From: 


Systems  Journal 

^  Voi.  33,  No.  1,  1994 


Adopting  Cleanroom 
software  engineering 
with  a  phased  approach 

by  P.  A.  Hausler 
R.  C.  Linger 
C.  J.  Trammell 


©  Copyright  1994  by  International  Business  Machines  Corporation.  See  individual  articles  for  copying  information.  Table  of  Contents 
page  may  be  freely  copied  and  distributed  in  any  form.  ISSN  18-8670.  Printed  in  U.S.A. 


Adopting  Cleanroom 
software  engineering 
with  a  phased  approach 


by  P.  A.  Hausler 
R.  C.  Linger 
C.  J.  Trammell 


Cleanroom  software  engineering  is  a  theory- 
based,  team-oriented  engineering  process  for 
developing  very  high  quality  software  under 
statistical  quality  control.  The  Cleanroom  process 
combines  formal  methods  of  object-based  box 
structure  specification  and  design,  function- 
theoretic  correctness  verification,  and  statistical 
usage  testing  for  reliability  certification  to  produce 
software  approaching  zero  defects.  Management  of 
the  Cleanroom  process  is  based  on  a  life  cycle  of 
development  and  certification  of  a  pipeline  of  user- 
function  increments  that  accumulate  into  the  final 
product.  Teams  in  IBM  and  other  organizations 
that  use  the  process  are  achieving  remarkable 
quality  results  with  high  productivity.  A  phased 
implementation  of  the  Cleanroom  process  enables 
quality  and  productivity  improvements  with  an 
increased  control  of  change.  An  introductory 
implementation  involves  the  application  of 
Cleanroom  principles  without  the  full  formality  of 
the  process;  full  implementation  involves  the 
comprehensive  use  of  formal  Cleanroom  methods; 
and  advanced  implementation  optimizes  the 
Drocess  through  additional  formal  methods,  reuse, 
and  continual  improvement.  The  AOEXPERT/MVS 
project,  the  largest  IBM  Cleanroom  effort  to  date, 
successfully  applied  an  introductory  level  of 
implementation.  This  paper  presents  both  the 
implementation  strategy  and  the  project  results. 


Zero  or  near-zero  defect  software  may  seem 
like  an  impossible  goal.  After  all,  the  expe¬ 
rience  in  the  first  generation  of  software  devel¬ 
opment  has  reinforced  the  seeming  inevitability 
of  errors  and  persistence  of  human  fallibility.  To¬ 
day,  however,  a  new  reality  in  software  develop¬ 
ment  belies  the  first-generation  experience. 


Cleanroom  software  engineering  teams  are  able 
to  develop  software  at  a  level  of  quality  and  re¬ 
liability  that  would  have  seemed  impossible  a  few 
years  ago,  and  are  doing  so  with  high  productiv¬ 
ity. 

Cleanroom  software  engineering  is  a  managerial 
and  technical  process  for  the  development  of  soft¬ 
ware  approaching  zero  defects  with  certified  re¬ 
liability.  1,2  The  Cleanroom  process  spans  the  en¬ 
tire  software  life  cycle;  it  provides  a  complete 
discipline  within  which  software  teams  can  plan, 
specify,  design,  verify,  code,  test,  and  certify 
software.  The  Cleanroom  approach  treats  soft¬ 
ware  development  as  an  engineering  process 
based  on  mathematical  foundations,  rather  than 
as  a  trial-and-error  programming  process,3*7  and 
is  intended  to  produce  software  with  error-free 
designs  and  failure-free  executions. 

In  traditional,  craft-based  software  development, 
errors  were  accepted  as  inevitable,  and  program¬ 
mers  were  encouraged  to  get  software  into  testing 
quickly  in  order  to  begin  debugging.  Programs 
were  subjected  to  unit  testing  and  debugging  by 
their  authors,  then  integrated  into  components, 
subsystems,  and  systems  for  more  debugging. 

^Copyright  1994  by  International  Business  Machines  Corpo¬ 
ration.  Copying  in  printed  form  for  private  use  is  permitted 
without  payment  of  royalty  provided  that  (1)  each  reproduc¬ 
tion  is  done  without  alteration  and  (2)  the  Journal  reference 
and  IBM  copyright  notice  are  included  on  the  first  page.  The 
title  and  abstract,  but  no  other  portions,  of  this  paper  may  be 
copied  or  distributed  royalty  free  without  further  permission 
by  computer-based  and  other  information-service  systems. 
Permission  to  republish  any  .other  portion  of  this  paper  must 
be  obtained  from  the  Editor! 
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Product  use  by  customers  resulted  in  still  more 
debugging  to  correct  errors  discovered  in  opera¬ 
tional  use.  The  most  virulent  errors  were  often  the 
result  of  fixes  to  other  errors,8  and  it  was  not 
unusual  for  software  products  to  reach  a  steady- 
state  error  population,  with  new  errors  introduced 
as  fast  as  old  ones  were  fixed.  Today,  however, 
craft-based  processes  that  depend  on  testing  and 
debugging  to  improve  reliability  are  understood 
to  be  inefficient  and  ineffective.  Experience  has 
shown  that  craft-based  processes  often  fail  to 
achieve  the  level  of  reliability  essential  to  a  society 
dependent  on  software  for  the  conduct  of  human 
affairs. 

In  the  Cleanroom  process,  correctness  is  built 
into  the  software  by  development  teams  through 
a  rigorous  engineering  process  of  specification, 
design,  and  verification.  The  more  powerful  pro¬ 
cess  of  team  correctness  verification  replaces  unit 
testing  and  debugging,  and  software  enters  sys¬ 
tem  testing  directly,  with  no  execution  by  devel¬ 
opment  teams.  All  errors  are  accounted  for  from 
first  execution  on,  with  no  private  unit  testing 
necessary  or  permitted.  Experience  shows  that 
Cleanroom  software  typically  enters  system  test¬ 
ing  approaching  zero  defects  and  occasionally  no 
defects  are  found  in  all  testing. 

Certification  (test)  teams  are  not  responsible  for 
“testing  in”  quality,  which  is  an  impossible  task, 
but  rather  for  certifying  the  quality  of  software 
with  respect  to  its  specification.  Certification  is 
performed  by  statistical  usage  testing  that  pro¬ 
duces  objective  assessments  of  product  quality. 
Errors,  if  any,  found  in  testing  are  returned  to  the 
development  team  for  correction.  If  the  quality  is 
not  acceptable,  the  software  is  removed  from 
testing  and  returned  to  the  development  team  for 
reverification. 

The  process  of  Cleanroom  development  and  cer¬ 
tification  is  carried  out  in  an  incremental  manner. 
System  functionality  grows  with  the  addition  of 
successive  code  increments  in  a  stepwise  integra¬ 
tion  process.  When  the  final  increment  is  added, 
the  system  is  complete.  Because  successive  in¬ 
crements  are  elaborating  the  top-down  design  of 
increments  already  in  execution,  interface  and 
design  errors  are  rare. 

This  paper  describes  key  Cleanroom  technologies 
and  summarizes  quality  results  achieved  by 
Cleanroom  teams.  It  presents  a  phased  approach 


to  Cleanroom  implementation  based  on  the  soft¬ 
ware  maturity  level  of  an  organization,  and  sum¬ 
marizes  the  results  of  a  substantial  IBM  Clean¬ 
room  project  (AOEXPERT/MVS*)  that  successfully 
applied  a  phased  approach. 

Cleanroom  perspectives 

The  Cleanroom  software  engineering  process 
evolved  from  concepts  developed  and  demon¬ 
strated  over  the  past  15  years  by  Harlan  Mills  and 
colleagues. 3-5,9  Cleanroom  practices  such  as  step¬ 
wise  refinement  of  procedure  and  object  hierar¬ 
chies,  team  verification  of  correctness,  and  sta¬ 
tistical  usage  testing,  have  been  successfully 
applied  in  commercial  and  governmental  soft¬ 
ware  projects  over  the  past  decade.  Such  prac¬ 
tices  may  not  be  the  rule  in  software  development 
today,  but  their  use  is  growing  as  evidence  of  their 
value  continues  to  accumulate.  In  many  cases, 
software  organizations  considering  a  transition  to 
the  Cleanroom  process  have  operational  prac¬ 
tices  in  place,  such  as  incremental  development, 
structured  programming,  and  team  reviews,  that 
support  Cleanroom  concepts.  There  are  only  a 
few  key  concepts  that  must  be  understood  and 
accepted  in  a  transition  to  the  Cleanroom  ap¬ 
proach.10 

Practice  based  on  theory.  To  be  effective,  any  en¬ 
gineering  discipline  must  be  based  on  sound  the¬ 
oretical  foundations.  Cleanroom  specification, 
design,  and  correctness  verification  practices  are 
based  on  function  theory,  whereby  programs  are 
treated  as  rules  for  mathematical  functions  sub¬ 
ject  to  stepwise  refinement  and  verification.4,5 
Cleanroom  testing  and  quality  certification  prac¬ 
tices  are  based  on  statistical  theory,  whereby  pro¬ 
gram  executions  are  treated  as  populations  sub¬ 
ject  to  usage-based,  stochastic  sampling  in  formal 
statistical  designs.3,6,11  These  theoretical  founda¬ 
tions  form  the  basis  of  a  comprehensive  engineer¬ 
ing  process  that  has  been  reduced  to  practice  for 
commercial  software  development.  A  growing 
number  of  successful,  real-world  Cleanroom 
projects  have  demonstrated  the  practicality  of 
these  methods. 

Experienced  Cleanroom  practitioners  and  educa¬ 
tors  have  developed  comprehensive  technology 
transfer  programs  based  on  readily  teachable, 
time-efficient  approaches  to  such  Cleanroom 
technologies  as  correctness  verification  and  sta¬ 
tistical  testing.  New  practitioners  will  find  that 
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processes  and  tools  exist  that  make  the  use  of 
these  Cleanroom  methods  highly  practical. 12 


Right  the  first  time.  A  primary  objective  of  the 
Cleanroom  process  is  to  prevent  errors,  rather  than 
accepting  and  accommodating  errors  through  insti¬ 
tutionalized  debugging  and  rework.  For  this  reason, 
Cleanroom  development  teams  do  not  unit  test  and 
debug  their  code.  Instead,  they  rely  on  rigorous 
methods  of  specification  and  design  combined  with 
team  correctness  verification.  These  Cleanroom 
development  practices,  based  on  mathematical 
foundations,  yield  quality  approaching  zero  defects 
prior  to  first  execution  by  certification  teams.  The 
purpose  of  testing  in  Cleanroom  is  the  certification 
of  software  quality  with  respect  to  specifications, 
not  the  attempt  to  “debug  in”  quality. 


Management  understanding  and  acceptance  of 
this  essential  point— that  quality  will  be  achieved 
by  design  and  verification  rather  than  by  testing — 
must  be  reflected  in  the  development  schedule. 
Time  spent  in  specification  and  design  phases  of 
a  Cleanroom  development  is  greater  than  in  tra¬ 
ditional  projects.  Time  spent  in  testing,  however, 
is  likely  to  be  less  than  traditionally  required.  The 
manager  who  wanted  to  start  coding  immediately 
because  of  the  large  amount  of  debugging  ex¬ 
pected  was  usually  right,  but  would  have  diffi¬ 
culty  becoming  part  of  a  Cleanroom  team. 


Quality  costs  less.  A  principal  justification  for  the 
Cleanroom  process  is  that  built-in  quality  lowers 
the  overall  cost  to  produce  and  maintain  a  prod¬ 
uct.  The  exponential  growth  in  the  cost  of  error 
correction  in  successive  life-cycle  phases  is  well 
known.  Errors  found  in  operational  use  by  cus¬ 
tomers  are  typically  several  orders  of  magnitude 
more  costly  to  correct  than  errors  found  in  the 
specification  phase.13  The  Cleanroom  name, 
taken  from  the  semiconductor  industry  where  a 
literal  cleanroom  exists  to  prevent  introduction  of 
defects  during  hardware  fabrication,  is  a  meta¬ 
phor  that  reflects  this  understanding  of  the  cost- 
effectiveness  of  error  prevention.  In  the  Clean¬ 
room  process,  incremental  development  and 
extensive  team  review  and  verification  permit  er¬ 
rors  to  be  detected  as  early  as  possible  in  the  life 
cycle.  By  reducing  the  cost  of  errors  during  de¬ 
velopment  and  the  incidence  of  failures  during 
operation,  the  overall  life-cycle  cost  of  Clean¬ 
room  software  can  be  expected  to  be  far  lower 
than  industry  averages.  For  example,  the  IBM 
COBOL  Structuring  Facility  product,  developed 


using  Cleanroom  techniques,  has  required  only  a 
small  fraction  of  its  maintenance  budget  to  be 
consumed  during  years  of  field  use. 

Cleanroom  project  schedules  have  equaled  or  im¬ 
proved  upon  traditional  development  sched¬ 
ules.14"16  In  fact,  productivity  improvements  of 
factors  ranging  from  one  and  one-half  to  five  over 


A  primary  objective 
of  the  Cleanroom  process 
is  to  prevent  errors. 


traditional  practices  have  been  observed. 15-18  Ex¬ 
perienced  Cleanroom  teams  become  remarkably 
efficient  at  writing  clear  specifications,  simplify¬ 
ing  and  restricting  designs  to  easily  verifiable  pat¬ 
terns,  and  performing  correctness  verification. 
Cleanroom  is  not  a  more  time-consuming  devel¬ 
opment  process,  but  it  does  place  greater  empha¬ 
sis  on  design  and  verification  to  avoid  waste  of 
resources  in  debugging  and  rework. 

Cleanroom  quality  results 

As  summarized  in  Table  1,  first-time  Cleanroom 
teams  in  IBM  and  other  industrial  and  governmen¬ 
tal  organizations  have  reported  data  on  close  to  a 
million  lines  of  Cleanroom-developed  software. 
The  code  exhibits  a  weighted  average  of  2.3  er¬ 
rors  per  thousand  lines  of  code  (errors/KLOC)  in 
testing.2,15"19  This  error  rate  represents  all  errors 
found  in  all  testing,  measured  from  first-ever  ex¬ 
ecution  through  test  completion.  That  is,  it  is  a 
measure  of  residual  errors  remaining  following 
correctness  verification  by  development  teams, 
who  do  not  execute  the  software.  The  projects 
represent  a  variety  of  environments,  including 
batch,  distributed,  cooperative,  and  real-time 
systems  and  system  parts,  and  a  variety  of  lan¬ 
guages,  including  microcode,  C,  C+  +  ,  jovial, 
FORTRAN,  and  PL/1. 

Traditionally  developed  software  does  not  un¬ 
dergo  correctness  verification,  but  rather  enters 
unit  testing  and  (debugging  directly,  followed  by 
more  debugging  in  function  and  system  testing 
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Table  1  Cleanroom  project  results 


Year 

Project 

Quality  and  Productivity 

1987 

IBM  Flight  Control: 

Helicopter  Avionics  System  Component 

33  KLOC  (JOVIAL) 

•  Certification  testing  failure  rate:  2.3  errors/KLOC 

•  Error-fix  reduced  5X 

•  Completed  ahead  of  schedule 

1988 

IBM  Cobol  Structuring  Facility:  Product  for 
automatically  restructuring  COBOL  programs 

85  KLOC  (PL/I) 

•  IBM’s  first  Cleanroom  product 

•  Certification  testing  failure  rate:  3.4  errors/KLOC 

•  Productivity  740  LOC/PM,  5X  improvement 

•  7  errors  in  first  3  years  of  use;  all  simple  fixes 

1989 

NASA  Satellite  Control  Project  1 

40  KLOC  (FORTRAN) 

•  Certification  testing  failure  rate:  4.5  errors/KLOC 

•  50%  improvement  in  quality 

•  Productivity  780  LOC/PM 

•  80%  improvement  in  productivity 

1990 

Martin  Marietta: 

Automated  documentation  system 

1.8  KLOC  (FOXBASE) 

•  First  compilation:  no  errors  found 

•  Certification  testing  failure  rate:  0.0  errors/KLOC 
(no  errors  found) 

1991 

IBM  System  Software 

First  increment  0.6  KLOC  (C) 

•  First  compilation:  no  errors  found 

•  Certification  testing  failure  rate:  0.0  errors/KLOC 
(no  errors  found) 

1991 

IBM  AOEXPERT/MVS™  Product 

107  KLOC  (mixed  languages) 

•  Testing  failure  rate:  2.6  errors/KLOC 

•  Productivity  486  LOC/PM 

•  No  operational  errors  from  Beta  test  sites 

1991 

IBM  Language  Product 

First  increment  21.9  KLOC  (PL/X) 

•  Testing  failure  rate:  2. 1  errors/KLOC 

1991 

IBM  Image  Product  Component 

3.5  KLOC  (C) 

•  First  compilation:  5  syntax  errors 

•  Certification  testing  failure  rate:  0.9  errors/KLOC 

1992 

IBM  Printer  Application 

First  increment  6.7  KLOC  (C) 

•  Certification  testing  failure  rate:  5.1  errors/KLOC 

1992 

IBM  Knowledge  Based  System  Application 

17.8  KLOC  (TIRS™) 

•  Testing  failure  rate:  3.5  errors/KLOC 

1992 

NASA  Satellite  Control  Projects  2  and  3 

170  KLOC  (FORTRAN) 

*  Testing  failure  rate:  4.2  errors/KLOC 

1993 

University  of  Tennessee:  Cleanroom  tool 

20  KLOC  (C) 

•  Certification  testing  failure  rate:  6.1  errors/KLOC 

1993 

IBM  3490E  Tape  Drive 

86  KLOC  (C) 

•  Certification  testing  failure  rate:  1.2  errors/KLOC 

1993 

IBM  Database  Transaction  Processor 

First  increment  2  L5  KLOC  (JOVIAL) 

•  Testing  failure  rate:  2.4  errors/KLOC 

•  No  design  errors,  all  simple  fixes 

1993 

IBM  LAN  Software 

First  increment  4.8  KLOC  (C) 

•  Testing  failure  rate:  0.8  errors/KLOC 

1993 

IBM  Workstation  Application  Component 

3.0  KLOC  (JOVIAL) 

•  Testing  failure  rate:  4.1  errors/KLOC 

1993 

Ericsson  Telecom  AB  Switching  Computer  OS32 
Operating  System 

350  KLOC  (PLEX,  C) 

•  Testing  failure  rate:  1  error/KLOC 

•  70%  improvement  in  development  productivity 

•  100%  improvement  in  testing  productivity 

NOTH:  All  testing  failure  rates  are  measured  from  first-ever  KEY :  KLOC  =  thousand  lines  of  code 

execution.  PM  =  person  month 

X  =  (mathematical)  times 
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following.  Measured  from  first  execution,  tradi¬ 
tional  software  typically  exhibits  25  to  35  or  more 
errors  per  thousand  lines  of  code.20  First-time 
Cleanroom  development  teams  can  produce  soft¬ 
ware  with  quality  levels  at  test  entry  at  least  an 
order  of  magnitude  better  than  traditionally  de¬ 
veloped  software.  The  following  summaries  of 
three  selected  projects  from  Table  1  illustrate  the 
results  achieved. 

IBM  COBOL  Structuring  Facility.  The  COBOL 
Structuring  Facility,  which  consisted  of  85  KLOC 
of  PL/I  code,  was  the  first  Cleanroom  product  in 
IBM.  It  employs  proprietary,  graph-theoretic  al¬ 
gorithms  to  automatically  transform  unstructured 
COBOL  programs  into  a  functionally  equivalent, 
structured  form  for  improved  maintainability.  Re¬ 
lentless  design  simplification  in  the  Cleanroom 
process  often  results  in  systems  that  are  small  for 
their  functionality.  For  example,  the  Cleanroom- 
developed  prototype  of  the  COBOL  Structuring 
Facility,  independently  estimated  at  100  KLOC, 
was  developed  using  just  20  KLOC. 

Comparable  to  a  COBOL  compiler  in  complexity, 
the  product  experienced  3.4  errors/KLOC  in  all 
statistical  testing,  measured  from  the  first  execu¬ 
tion.  Six  months  of  intensive  beta  testing  at  a  ma¬ 
jor  aerospace  corporation  resulted  in  no  func¬ 
tional  equivalence  errors  ever  found.21  Just  seven 
minor  errors  were  reported  in  the  first  three  years 
of  field  use,  requiring  only  a  small  fraction  of  the 
maintenance  budget  associated  with  traditionally 
developed  products  of  similar  size  and  complex¬ 
ity.  The  product  was  developed  and  certified  by 
a  team  averaging  six  members,  with  productivity 
five  times  the  IBM  averages. 16 

IBM  3490E  tape  drive.  The  3490E  tape  drive  is  a 
real-time,  embedded  software  system  developed 
by  a  five-person  team  in  three  increments  of  C 
design  with  a  code  total  of  86  KLOC.  It  provides 
high-performance  tape  cartridge  support  through 
a  multiple  processor  bus  architecture  that  pro¬ 
cesses  multiple  real-time  input  and  output  data 
streams.  The  product  experienced  1.2  errors/ 
KLOC  in  all  statistical  testing.  To  meet  an  urgent 
business  need,  the  third  increment  was  shipped 
straight  from  development  to  the  hardware  and 
software  integration  team  with  no  testing  what¬ 
soever.  Customer  evaluation  testing  with  live 
data  by  the  integration  team  resulted  in  no  errors 
being  found. 


In  a  comparison  experiment,  the  project  team 
subjected  a  selected  module  to  both  unit  testing 
and  correctness  verification.  Development  of  ex¬ 
ecution  scaffolding,  definition  and  execution  of 
test  cases,  and  checking  of  results  required  one- 
and  one-half  person-weeks  of  effort  and  resulted 
in  the  detection  of  seven  errors.  Correctness  ver¬ 
ification  of  the  same  program  by  the  development 
team  required  one  and  one-half  hours,  and  re¬ 
sulted  in  the  detection  of  the  same  seven  errors, 
plus  three  additional  errors. 1 

Ericsson  OS32  operating  system.  Ellemtel  Tele¬ 
communications  Systems  Laboratories  is  com¬ 
pleting  a  350  KLOC  operating  system  for  a  new 
family  of  switching  computers  for  Ericsson  Tele¬ 
com  AB.  The  code  is  written  in  PLEX  and  C.  The 
73-person,  33-month  Cleanroom  project  experi¬ 
enced  productivity  improvements  of  70  percent 
and  100  percent  in  development  and  testing,  re¬ 
spectively,  and  the  product  averaged  under  one 
error/KLOC  in  all  testing.  Project  management  re¬ 
ported  that  an  average  of  less  than  one  person- 
hour  was  required  to  detect  an  error  in  team  re¬ 
views,  compared  to  an  average  of  17.5  person- 
hours  to  detect  an  error  in  testing.  The  project 
allocated  two  days  per  week  to  prepare  and  con¬ 
duct  team  reviews.  The  product  team  was  hon¬ 
ored  by  Ericsson  as  the  single  project  that  had 
contributed  the  most  to  the  company  in  1993. 18 

Cleanroom  technologies 

In  the  Cleanroom  process,  the  objective  of  the 
development  team  is  to  deliver  software  to  the 
test  team  that  approaches  zero  defects;  the  ob¬ 
jective  of  the  test  team  is  to  scientifically  certify 
the  quality  of  software,  not  to  attempt  to  “test  in” 
quality.  These  objectives  are  achieved  through 
management  and  technical  practices  based  on  the 
technologies  of  incremental  development,  box 
structure  specification  and  design,  correctness 
verification,  and  statistical  quality  certification. 

Incremental  development.  Management  planning 
and  control  in  Cleanroom  is  based  on  develop¬ 
ment  and  certification  of  a  pipeline  of  increments 
that  represent  operational  user  function,  accumu¬ 
late  top-down  into' the  final  product,  and  execute 
in  the  system  environment.22  Following  specifi¬ 
cation  of  required  external  system  behavior,  an 
incremental  development  plan  is  created  to  define 
schedules,  resources,  and  functional  content  of  a 
series  of  code  increments  to  be  developed  and 
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certified.  The  initial  increment  contains  stubs 
(small  placeholder  programs)  that  stand  in  for 
later  increments  and  permit  early  execution  of  the 
code.  The  ultimate  functionality  of  the  code  that 
will  replace  the  stubs  is  fully  defined  in  subspeci- 


When  the  final  increment 
is  integrated,  the 
system  is  complete. 


fications  for  team  verification  of  each  increment 
prior  to  testing.  As  incremental  development 
progresses,  stubs  are  replaced  by  corresponding 
code  increments,  possibly  containing  stubs  of 
their  own,  in  a  stepwise  system  integration  pro¬ 
cess.  When  the  final  increment  is  integrated,  the 
system  is  complete  and  no  stubs  remain. 

As  each  increment  is  integrated,  the  evolving  sys¬ 
tem  of  increments  undergoes  a  new  step  in  sta¬ 
tistical  usage  testing  for  quality  certification.  Sta¬ 
tistical  measures  of  quality  provide  feedback  for 
reinforcement  or  improvement  of  the  develop¬ 
ment  process  as  necessary.  Early  increments  can 
serve  as  system  prototypes,  providing  an  oppor¬ 
tunity  to  elicit  feedback  from  customers  to  vali¬ 
date  requirements  and  functionality.  As  inevita¬ 
ble  changes  occur,  incremental  development 
provides  a  framework  for  revising  schedules,  re¬ 
sources,  and  function,  and  permits  changes  to  be 
incorporated  in  a  systematic  manner. 

Box  structure  specification  and  design.  Box  struc¬ 
tures  provide  a  stepwise  refinement  and  verifica¬ 
tion  process  based  on  black  box ,  state  box ,  and 
clear  box  forms  for  defining  system  behavior  and 
deriving  and  connecting  objects  comprising  a  sys¬ 
tem  architecture.3,23  Boxes  are  object-based,  and 
the  box  structure  process  provides  a  systematic 
means  for  developing  object-based  systems.24 
Specifically,  the  black  box  form  is  a  specification 
of  required  behavior  of  a  system  or  system  part  in 
all  circumstances  of  use,  defined  in  terms  of  stim¬ 
uli,  responses,  and  transition  rules  that  map  stim¬ 
ulus  histories  to  responses.  The  state  box  form  is 
refined  from  and  verified  against  the  black  box, 
and  defines  encapsulated  state  data  required  to 


satisfy  black  box  behavior.  The  clear  box  form  is 
refined  from  and  verified  against  the  state  box, 
and  defines  procedural  design  of  services  on  state 
data  to  satisfy  black  box  behavior,  often  intro¬ 
ducing  new  black  boxes  at  the  next  level  of  re¬ 
finement.  New  black  boxes  (specifications)  are 
similarly  refined  into  state  boxes  (state  designs) 
and  clear  boxes  (procedure  designs),  continuing 
in  this  manner  until  no  new  black  boxes  are  re¬ 
quired.  Specification  and  design  steps  are  inter¬ 
leaved  in  a  seamless,  integrated  hierarchy  afford¬ 
ing  complete  verifiability  and  traceability. 

Box  structures  isolate  and  separate  the  creative 
definition  of  behavior,  data,  and  procedures  at 
each  level  of  refinement.  They  incorporate  the 
essential  property  of  referential  transparency , 
such  that  the  information  content  of  an  abstrac¬ 
tion,  for  example,  a  black  box,  is  sufficient  to 
define  and  verify  its  refinement  into  state  and 
clear  box  forms  without  reference  to  other  spec¬ 
ification  parts.  Referential  transparency  is  crucial 
to  maintaining  intellectual  control  in  complex  sys¬ 
tem  developments.  Box-structured  systems  are 
developed  as  usage  hierarchies  of  boxes,  where 
each  box  provides  services  on  encapsulated  state 
data,  and  where  its  services  may  be  used  and  re¬ 
used  in  many  places  in  the  hierarchy  as  required. 
Box-structured  systems  are  developed  according 
to  the  following  principles:25  (1)  all  data  to  be  de¬ 
fined  and  retained  in  a  design  are  encapsulated  in 
boxes,  (2)  all  processing  is  defined  by  sequential 
and  concurrent  use  of  boxes,  and  (3)  each  use  of 
a  box  occupies  a  distinct  place  in  the  usage  hier¬ 
archy  of  the  system.  Clear  boxes  play  an  impor¬ 
tant  role  in  the  hierarchy  by  defining  and  control¬ 
ling  the  correct  operation  of  box  services  at  the 
next  level  of  refinement. 

Correctness  verification.  As  noted,  in  the  Clean- 
room  process,  verification  of  program  correct¬ 
ness  in  team  reviews  replaces  private  unit  testing 
and  debugging  by  individuals.  Debugging  is  an 
inefficient  and  error-prone  process  that  under¬ 
mines  the  mental  discipline  and  concentration 
that  can  achieve  zero  defects.  The  intellectual 
control  of  software  development  afforded  by 
team  verification  is  a  strong  incentive  for  the  pro¬ 
hibition  against  unit  testing.  “No  unit  testing” 
does  not,  however,  mean  “no  use  of  the  ma¬ 
chine.”  It  is  essential  to  use  the  machine  for  ex¬ 
perimentation,  to  evaluate  algorithms,  to  bench¬ 
mark  performance,  and  to  understand  and 
document  the  semantics  of  interfacing  software. 
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These  exploratory  activities  are  entirely  consis¬ 
tent  with  the  Cleanroom  objective  of  software 
that  is  correct  by  design. 

Elimination  of  unit  testing  motivates  tremendous 
determination  in  developers  to  ensure  that  the 
code  they  deliver  for  independent  testing  is  error- 
free  on  first  execution.  But  there  is  a  deeper  rea¬ 
son  to  adopt  correctness  verification— it  is  more 
efficient  and  effective  than  unit  testing.  Programs 
of  any  size  can  contain  an  essentially  infinite  num¬ 
ber  of  possible  execution  paths  and  states,  but 
only  a  minute  fraction  of  those  can  be  exercised 
in  unit  testing.  Correctness  verification,  however, 
reduces  the  verification  of  programs  to  a  finite  and 
complete  process. 

In  more  detail,  all  clear  box  programs  are 
composed  of  nested  and  sequenced  control 
structures,  such  as  sequence,  IF-THEN-ELSE, 
WHILE-DO,  and  their  variants.  Each  such  control 
structure  is  a  rule  for  a  mathematical  function, 
that  is,  a  mapping  from  a  domain  or  initial  state  to 
a  range  or  final  state.  The  function  mapping  car¬ 
ried  out  by  each  control  structure  can  be  docu¬ 
mented  in  the  design  as  an  intended  function.  For 
correctness,  each  control  structure  must  imple¬ 
ment  the  precise  mapping  defined  by  its  intended 
function.  The  Correctness  Theorem4  shows  that 
verification  of  sequence,  IF-THEN-ELSE,  and 
WHILE-DO  structures  requires  checking  exactly 
one,  two,  and  three  correctness  conditions,  re¬ 
spectively.  While  programs  can  exhibit  an  essen¬ 
tially  infinite  number  of  execution  paths  and 
states,  they  are  composed  of  a  finite  number  of 
control  structures,  and  their  verification  can  be 
carried  out  in  a  finite  number  of  steps  by  checking 
each  correctness  condition  in  team  reviews.  Fur¬ 
thermore,  verification  is  complete,  that  is,  it  deals 
with  all  possible  program  behavior  at  each  level  of 
refinement.  The  verification  process  defined  by 
the  Correctness  Theorem  accounts  for  all  possi¬ 
ble  mappings  from  the  domain  to  the  range  of 
each  control  structure,  not  just  a  handful  of  map¬ 
pings  exercised  by  particular  unit  tests.  For  these 
reasons,  verification  far  surpasses  unit  testing  in 
effectiveness. 

Statistical  quality  certification.  In  the  Cleanroom 
process,  statistical  usage  testing  for  certification 
replaces  coverage  testing  for  debugging.  Testing 
is  carried  out  by  the  certification  team  based  on 
anticipated  usage  by  customers.  Usage  probabil¬ 
ity  distributions  are  developed  to  define  system 


inputs  for  all  aspects  of  usage,  including  nominal 
scenarios  as  well  as  error  and  stress  situations. 
The  distributions  can  be  organized  into  probabi- 


Debugging  is  an 
inefficient  and 
error-prone  process. 


listic  state  transition  matrices  or  formal  gram¬ 
mars.  Test  cases  are  generated  based  on  random 
sampling  of  usage  distributions.  The  correct  out¬ 
put  for  each  test  input  is  specified  with  reference 
to  an  oracle,  that  is,  an  independent  authority  on 
correctness,  typically  the  software  specification. 
System  reliability  is  predicted  based  on  analysis 
of  test  results  by  a  formal  reliability  model,  and 
the  development  process  for  each  increment  is 
evaluated  based  on  the  extent  to  which  the  reli¬ 
ability  results  attained  objectives.  In  effect,  sta¬ 
tistical  usage  testing  is  based  on  a  formal  statis¬ 
tical  design,  from  which  statistical  inferences 
of  software  quality  and  reliability  can  be  de¬ 
rived.3’11’26 

Coverage  testing  can  provide  no  more  than  an¬ 
ecdotal  evidence  of  reliability.  Thus,  if  many  er¬ 
rors  are  found,  does  that  that  mean  that  the  code 
is  of  poor  quality  and  many  errors  remain,  or  that 
most  of  the  errors  have  been  discovered?  Con¬ 
versely,  if  few  errors  are  found,  does  that  mean 
that  the  code  is  of  good  quality,  or  that  the  testing 
process  is  ineffective?  Statistical  testing  provides 
scientifically  valid  measures  of  reliability,  such  as 
mean-time-to-failure  (mttf),  as  a  basis  for  objec¬ 
tive  management  decision-making  regarding  soft¬ 
ware  and  development  process  quality. 

Empirical  studies  have  demonstrated  enormous 
variation  in  the  failure  rates  of  errors  in  opera¬ 
tional  use.8  Correcting  high-failure-rate  errors 
has  a  substantial  effect  on  MTTF,  while  correcting 
low-failure-rate  errors  hardly  influences  MTTF  at 
all.  Because  usage-based  testing  exercises  soft¬ 
ware  the  way  users  intend  to  use  it,  high-fre¬ 
quency,  virulent 'errors  tend  to  be  found  early  in 
testing.  For  this  reason,  statistical  usage  testing  is 
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more  effective  at  improving  software  reliability 
than  is  coverage  testing.  Statistical  testing  also 
provides  new  management  flexibility  to  certify 
software  quality  for  varying  conditions  of  use  and 
stress,  by  developing  special  usage  probability 
distributions  for  such  situations.  For  example, 
the  reliability  of  infrequently  used  functions  with 
severe  consequences  of  failure  can  be  indepen¬ 
dently  measured  and  certified. 

Adopting  the  Cleanroom  process 

Rigorous  and  complete  Cleanroom  implementa¬ 
tion  permits  development  of  very  high  quality 
software  with  scientific  certification  of  reliability. 
However,  substantial  gains  in  quality  and  pro¬ 
ductivity  have  also  occurred  in  partial  Cleanroom 
implementations. 15,18  Evidence  suggests  that  a 
phased  approach  to  implementation  can  produce 
concrete  benefits  and  afford  increased  manage¬ 
ment  control.  The  phased  approach,  combined 
with  initial  Cleanroom  use  on  selected  demon¬ 
stration  projects,  provides  a  systematic  manage¬ 
ment  process  for  reducing  risk  in  technology 
transfer.  Three  implementation  phases  can  be  de¬ 
fined  and  sequenced  in  a  systematic  technology 
transfer  process.  The  idea  is  to  first  introduce  fun¬ 
damental  Cleanroom  principles  and  several  key 
technologies  in  an  introductory  implementation . 
As  team  experience  and  confidence  grows,  in¬ 
creased  precision  and  rigor  can  be  achieved  in  a 
full  implementation  of  Cleanroom  technology.  Fi¬ 
nally,  an  advanced  implementation  can  be  intro¬ 
duced  to  optimize  the  Cleanroom  process.  Of 
course,  a  particular  Cleanroom  implementation 
can  combine  elements  from  various  phases  as 
necessary  and  appropriate  for  the  project  envi¬ 
ronment. 

Introductory  implementation.  Key  aspects  of  an  in¬ 
troductory  implementation  are  summarized  in  the 
first  row  of  Table  2.  The  fundamental  idea  is  to  shift 
from  craft-based  to  engineering-based  processes. 
The  development  objective  shifts  from  defect  cor¬ 
rection  in  unit  testing  to  defect  prevention  in  spec¬ 
ification,  design,  and  verification.  As  experience 
grows,  developers  learn  they  can  write  software 
that  is  right  the  first  time,  and  a  psychological 
change  occurs,  from  expecting  errors  to  expecting 
correctness.  At  the  same  time,  the  testing  objective 
shifts  from  debugging  in  coverage  testing  to  reli¬ 
ability  certification  in  usage  testing.  Because  Clean¬ 
room  code  is  of  high  quality  at  first  execution, 
testers  learn  that  little  debugging  is  required,  and 


they  can  concentrate  on  evaluating  quality.  A  man¬ 
agement  opportunity  exists  to  leverage  these  tech¬ 
nology  shifts  to  develop  systems  on  schedule  with 
substantial  improvement  in  quality  and  reduction  in 
life-cycle  costs. 

All  development  and  testing  is  accomplished  by 
small  teams.  Team  operations  provide  opportu¬ 
nities  for  cross-training  and  a  ready  forum  for  dis¬ 
cussion,  review,  and  improvement.  All  work 
products  undergo  a  team-based  peer  review  to 
ensure  the  highest  level  of  quality.  The  size  and 
number  of  teams  varies  according  to  resource 
availability,  skill  levels,  and  project  size  and  com¬ 
plexity.  Teams  are  organized  during  project  plan¬ 
ning  and  their  membership  should  remain  stable 
throughout  development.  Cooperative  team  be¬ 
havior  that  leverages  individual  expertise  is  a  key 
factor  in  successful  Cleanroom  operations. 

In  any  Cleanroom  implementation,  zero-defect 
software  is  an  explicit  design  goal,  and  measured 
performance  at  a  target  level  is  an  explicit  reli¬ 
ability  goal.  The  Cleanroom  practices  necessary 
to  achieve  these  objectives  require  substantial 
management  commitment.  Because  compro¬ 
mises  in  process  inevitably  lead  to  compromises 
in  quality,  it  is  crucial  for  managers  to  understand 
Cleanroom  fundamentals — the  philosophy,  pro¬ 
cess,  and  milestones —  and  demonstrate  unequiv¬ 
ocal  support.  Management  commitment  is  essen¬ 
tial  to  successful  introduction  of  the  Cleanroom 
process. 

A  key  aspect  of  customer  interaction  is  to  shift 
from  a  technology-driven  to  a  customer-driven 
approach,  whereby  system  functional  and  usage 
requirements  are  subject  to  extensive  analysis 
and  review  with  customers  to  clearly  understand 
their  needs.  Maintaining  customer  involvement  in 
specification  and  certification  helps  avoid  devel¬ 
oping  a  system  that  approaches  zero  defects  but 
provides  the  wrong  functionality  for  the  user. 

Unlike  the  traditional  life  cycle  of  sequential 
phases,  the  Cleanroom  life  cycle  is  based  on  in¬ 
cremental  development.  In  an  introductory  im¬ 
plementation,  a  project  is  scheduled  and  managed 
as  a  pipeline  of  increments  for  development  and 
testing.  Functional  content  and  sequencing  of  in¬ 
crements  is  typically  based  on  a  natural  subdivi¬ 
sion  of  system  functions  and  their  expected  us¬ 
age.  Successive  increments  should  implement 
user  function,  execute  in  the  system  environ- 
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ment,  and  accumulate  top  down  into  the  final 
product.  This  incremental  strategy  supports  test¬ 
ing  throughout  development  rather  than  at  com¬ 
pletion.  It  also  integrates  system  increments  in 


Management  commitment  is 
essential  to  successful 
introduction. 


multiple  steps  across  the  life  cycle,  to  avoid  risks 
of  single-step  integration  of  all  system  compo¬ 
nents  late  in  a  project  when  little  time  or  re¬ 
sources  remain  to  deal  with  unforeseen  problems. 

In  an  introductory  implementation,  a  black  box 
specification  is  written  that  precisely  defines  re¬ 
quired  system  functionality  in  terms  of  inputs, 
outputs,  and  behavior  in  all  possible  circum¬ 
stances  of  use,  including  correct  and  incorrect 
use.  The  specification  focuses  on  required  system 
behavior  from  the  user’s  viewpoint  and  does  not 
describe  implementation  details.  At  this  level, 
specifications  are  generally  expressed  in  an  outer 
syntax  of  specification  structures,  such  as  tabular 
formats  or  variants  of  Box  Description  Language 
(bdl),5  and  an  inner  syntax  of  natural  language. 
Cleanroom  specifications  are  important  working 
documents  that  drive  design  and  certification  ac¬ 
tivities,  and  they  must  be  kept  current  for  effec¬ 
tive  team  operations.  Definition  of  system  user’s 
guides  is  initiated  in  parallel  with  specifications, 
for  elaboration  and  refinement  throughout  the  de¬ 
velopment. 

In  the  design  process  of  an  introductory  imple¬ 
mentation,  state  and  clear  box  concepts  are  im¬ 
plemented  using  sound  software  engineering 
practices,  including  stepwise  refinement,  struc¬ 
tured  programming,  modular  design,  information 
hiding,  and  data  abstraction.  Successive  incre¬ 
ments  are  specified  and  designed  top-down 
through  stepwise  refinement,  with  frequent  team 
review  and  discussion  of  design  strategies.8  Step¬ 
wise  refinement  requires  substantial  look-ahead 
and  analysis,  as  successive  design  versions  are 
developed  and  revised.  In  this  process,  a  relent¬ 


less  team  drive  for  design  simplification  can  result 
in  substantial  reductions  in  the  size  and  complex¬ 
ity  of  systems,  for  more  efficient  correctness  ver¬ 
ification  and  subsequent  maintenance. 

Design  with  intended  functions  is  a  fundamental 
practice  at  the  introductory  level.  High-level  in¬ 
tended  functions  originate  in  system  specifica¬ 
tions,  and  are  refined  into  control  structures  and 
new  intended  functions.  Expressed  primarily  in 
natural  language,  intended  functions  are  recorded 
as  comments  attached  to  key  control  structures  in 
designs.  Intended  functions  precisely  define  re¬ 
quired  behavior  of  their  control  structure  refine¬ 
ments.  Behavior  is  defined  in  functional,  non¬ 
procedural  descriptions  of  the  derivation  of 
output  data  from  input  data.  Intended  function 
refinements  are  expressed  in  a  restricted  set  of 
single-entry,  single-exit  control  structures  with 
no  side  effects,  such  as  sequence,  IF-THEN-ELSE, 
while-do,  and  their  variants.  Each  control  struc¬ 
ture  may  contain  additional  intended  functions 
for  further  refinement.  This  stepwise  specifica¬ 
tion  and  design  process  continues  until  no  further 
intended  functions  remain  to  be  elaborated.  In¬ 
tended  functions  provide  a  precise  road  map  for 
designers  in  refining  design  structures,  and  are 
essential  to  team  verification  reviews. 

The  last  intellectual  pass  through  a  design  occurs 
in  team-based  correctness  verification,  another 
fundamental  practice  in  an  introductory  imple¬ 
mentation.  At  the  design  level,  verification  re¬ 
views  prove  correctness  of  program  control 
structures,  unlike  traditional  code  inspections 
that  trace  program  flow  paths  to  look  for  errors. 
The  verification  process  is  based  on  reading  and 
abstracting  the  functionality  of  control  structures 
in  designs  and  comparing  the  abstractions  with 
specified  intended  functions  to  assess  correct¬ 
ness.  Team  members  read,  discuss,  evaluate,  and 
indicate  agreement  (or  not)  that  designs  are  cor¬ 
rect  with  respect  to  their  intended  behavior.  If 
changes  are  required,  the  team  must  review  and 
verify  the  modifications  before  the  designs  can  be 
considered  finished.  Verification  reviews  provide 
team  members  with  deep  understandings  of  de¬ 
signs  and  their  correctness  arguments.  Reviews 
are  conducted  with  the  understanding  that  the  en¬ 
tire  team  is  responsible  for  correctness.  Ultimate 
successes  are  team  successes,  and  failures  are 
team  failures.  All  specifications  and  designs  are 
subject  to  team  review,  without  exception.  Fol- 
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Table  2  A  phased  implementation  for  Cleanroom  practice 


Cleanroom 
— ^Practice 
Implementation^ 

Management 

and 

Team  Operations 

Customer 

Interaction 

Incremental 

Development 

System 

Specification 

Introductory 

Implementation 

•  Document  an  introductory 
Cleanroom  process. 

•  Shift  from  craft-based  to 
engineering-based  processes. 

•  Shift  from  defect  correction 
in  unit  testing  to  defect 
prevention  in  specification, 
design,  and  verification. 

•  Shift  from  debugging  in 
coverage  testing  to  quality 
certification  in  usage  testing. 

•  Shift  from  individual  to  small 
team  operations  with  team 
review  of  all  work  products. 

•  Establish  Cleanroom  projects 
and  provide  commitment, 
education,  and  recognition 
to  teams. 

•  Develop  to  schedule  with 
substantial  quality 
improvement  and  life  cycle 
cost  reduction. 

•  Shift  from  technology- 
driven  to  customer- 
driven  development 

•  Analyze  and  clarify 
functional  requirements 
with  customers  to 
develop  functional 
specifications. 

•  Analyze  and  clarify 
usage  requirements  with 
customers  to  develop 
usage  specifications. 

•  Review  and  validate 
functional  and  usage 
specifications  with 
customers. 

•  Revise  functional  and 
usage  specifications  as 
necessary  for  changing 
requirements. 

•  Shift  from  a 
sequential 
(waterfall)  to  an 
incremental  process. 

•  Define  increments 
that  implement  user 
function,  execute  in 
the  system 
environment,  and 
accumulate  top  down 
into  the  final  product. 

•  Define  and  evolve  an 
incremental 
development  plan  for 
schedules,  resources, 
and  increment  content 

•  Carry  out  scheduled 
incremental 
development  and 
testing  with  stepwise 
integration  of 
increments. 

•  Shift  from  informal, 
throwaway  specifica¬ 
tions  to  precise, 
working  specifications 
kept  current  through 
the  project  lifecycle. 

•  Define  specifications 
of  system  boundaries, 
interfaces,  and  required 
external  behavior  in  all 
possible  circumstances 
of  use,  including  correct 
and  incorrect  use. 

•  Express  specifications 
in  systematic  forms 
such  as  tables  that 
define  required  behavior 
in  natural  language. 

•  Develop  and  evolve 
system  user’s  guides  in 
parallel  with 
specifications. 

Full 

Implementation 

•  Document  a  full  Cleanroom 
process. 

•  Increase  development  rigor 
with  box  structure 
specification,  design,  and 
correctness  verification. 

•  Increase  testing  rigor 
with  scientific  measures  of 
reliability. 

•  Establish  larger  Cleanroom 
projects  as  teams  of  small 
teams  with  experienced 
leaders  from  previous 
projects. 

•  Develop  to  schedule  with 
substantial  quality  and 
productivity  improvement 
and  life  cycle  cost  reduction. 

•  Educate  customers  in 
Cleanroom  to  increase 
value,  cooperation, 
and  responsiveness  to 
customer  needs. 

•  Review  black  box 
functional  specifications 
with  customers  to 
support  increased  rigor 
in  specification. 

•  Review  usage 
specifications  with 
customers  to  support 
increased  rigor  in 
statistical  usage  testing. 

•  Provide  customers 
with  prototypes 
and  accumulating 
increments  for 
evaluation  and  feedback. 

•  Define  increments  to 
incorporate  early 
availability  of 
important  functions 
for  customer  feedback 
and  use. 

•  Rapidly  revise 
incremental  plans  for 
new  requirements 
and  actual  team 
performance,  and 
respond  to  schedule 
and  budget  changes. 

•  Develop  prototypes 
as  necessary  to 
validate  customer 
requirements 

and  operating 

environment 

characteristics. 

•  Define  black  box 
specifications  in 
systematic  structures 
such  as  transition 
tables  expressed  in 
conditional  rules 
and  precise 
natural  language. 

Advanced 

Implementation 

•  Document  an  advanced 
Cleanroom  process. 

•  Establish  a  Cleanroom 

Center  of  Competency  to 
monitor  Cleanroom 
technology  and  train  and 
consult  with  teams. 

•  Establish  Cleanroom  projects 
across  die  organization  led 
by  experienced  Cleanroom 
practitioners. 

•  Develop  to  schedule  with 
substantial  quality  and 
productivity  improvements 
and  life  cycle  cost  reduction, 
even  in  emergency  and 
adverse  circumstances. 

•  Assist  customers  in 
leveraging  the  quality 
of  Cleanroom- 
developed  software  for 
competitive  advantage. 

•  Contract  with  customer 
for  reliability  warranties 
based  on  certification 
with  agreed  usage 
distributions  and 
reliability  models. 

•  Establish  cooperative 
processes  with 
customers  for  recording 
operational  system 
usage  to  calibrate  and 
improve  reliability 
certification. 

•  Incorporate 
comprehensive 
reuse  analysis  and 
reliability  planning 
in  incremental 
development  plans. 

•  Plan  increment 
content  to  manage 
project  risk  by  early 
development  of 
interface 

dependencies,  critical 
functions,  and 
performance-sensitive 
processes. 

•  Incorporate  advances 
in  formal  specification 
methods  into  local 
practices. 

•  Develop  guidelines  for 
specification  formats 
and  conventions  based 
on  team  experience. 

•  Apply  mathematical 
techniques  in  black 
box  specifications 
to  define  complex 
behavior  with  precision. 

•  Express  black  box 
specifications  where 
appropriate  with 
specification  functions 
and  abstract  models. 

•  Develop  a  specification 
review  protocol  for 
team  reviews  based 
on  team  experience. 
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System  Design 
and 

Implementation 

Correctness 

Verification 

Statistical  Testing 
and 

Reliability  Certification 

Process 

Improvement 

•  Shift  from  programming  by 
aggregation  of  statements  to 
design  by  stepwise  refinement 
of  specifications. 

•  Refine  specifications  into 
structured,  modular  designs 
using  good  software  engineering 
practices  with  substantial  look 
ahead  and  analysis. 

•  Express  designs  in  control 
structures  and  case-structured 
intended  functions  expressed  in 
natural  language. 

•  Conduct  frequent  team 
development  reviews  to 
communicate,  simplify,  and 
improve  evolving  designs. 

•  Conduct  execution  experiments 
to  document  the  system 
environment  and  semantics  of 
interfacing  software. 

•  Shift  from  unit  testing  by  ( 
individuals  to  correctness 
verification  by  teams. 

•  Shift  from  path  tracing  in 
code  inspections  to 
functional  analysis  in 
verification  reviews. 

•  Conduct  demonstration 
verification  reviews  to  set 
expectations  and  train 
teams. 

•  Verify  all  control 
structures  in  team  reviews 
by  reading,  function 
abstraction,  and 
comparison  to  intended 
functions. 

•  Verify  all  design  changes 

in  team  reviews  and 
deliver  verified 
increments  to  testing  for 
first  execution.  1 

•  Shift  from  coverage  testing 
to  usage  testing. 

•  Define  high-level  usage 
distributions  in  systematic 
structures  such  as  hierarchical 
decision  trees. 

•  Develop/acquire  test  cases 
from  a  user  perspective 
based  on  system  specifications 
and  usage  distributions. 

•  Evaluate  quality  of  each 
increment  through  analysis 
of  measures  such  as  failure 
rates  and  severity  levels. 

•  Return  low-quality  increments 
to  development  for  additional 
design  and  reverification. 

•  Shift  from  informal  review 
of  lessons  learned  to  a 
systematic,  documented 
improvement  process. 

•  Measure  team  productivity, 
quality,  and  cost, 

and  analyze  for  process 
improvements. 

•  Document  improvements 
to  the  introductory 
implementation  based  on 
lessons  learned  from  each 
increment. 

•  Improve  or  sustain  the 
development  process  based 
on  quality  results  of 
increment  testing. 

•  Assess  customer  satisfaction 
with  Cleanroom-developed 
systems  for  process 
improvements. 

•  Refine  black  boxes 
(specifications)  into  state  boxes 
(data  designs)  and  state  boxes 
into  clear  boxes  (procedure 
designs)  and  new  black  boxes. 

•  Define  state  boxes  in  data 
designs  and  systematic  structures 
such  as  transition  tables 
expressed  in  conditional  rules 
and  precise  natural  language. 

•  Define  clear  boxes  in  control 
structures  and  intended  functions 
expressed  in  conditional  rules 
and  precise  natural  language. 

•  Encapsulate  system  data  in 
boxes  and  define  processing 
by  use  of  box  services. 

•  Identify  opportunities  for  reuse 
of  system  components. 

•  Improve  introductory 
practices  through 
increased  precision  and 
formality  in  verification 
reviews. 

•  Improve  verification  by 
introducing  mental  proofs 
of  correctness  based  on 
box  structure  theory  and 
Correctness  Theorem 
correctness  conditions. 

•  Document  and  reuse 
proof  arguments  for 
recurring  design  patterns. 

•  Simplify  and  standardize 
designs  where  possible  to 
reduce  proof  reasoning. 

•  Establish  reliability  targets 
and  conduct  statistical 
usage  testing  for  reliability 
certification. 

•  Define  usage  probability 
distributions  for  all 
circumstances  of  use  in 
formal  grammers  or  state 
transition  matrices. 

•  Define  alternative 
distributions  for  special 
environments  and  critical 
and  unusual  usage. 

•  Use  automated  generators 

to  create  test  cases  randomized 
against  usage  probability 
distributions. 

•  Use  reliability  models  to 
produce  statistical 
reliability  measures  based 
on  analysis  of  test  results. 

•  Document  improvements  to 
the  full  implementation 
based  on  team  decisions  in 
process  reviews  after  each 
increment. 

•  Use  baseline  measurements 
from  introductory  projects 
to  set  quality  and 
productivity  objectives. 

•  Improve  or  sustain  the 
development  process  based 
on  reliability  measurements 
of  each  increment. 

•  Conduct  causal  analysis  of 
failures  found  in  testing  and 
use  to  identify  process  areas 
for  improvement. 

•  Conduct  surveys  of  customer 
satisfaction  with  Cleanroom- 
developed  systems  for 
process  improvement. 

•  Incorporate  advances  in  formal 
design  methods  into  local 
practices. 

•  Use  box  structures  to  document 
the  precise  semantics  of 
interfacing  software. 

•  Develop  guidelines  for  design 
formats  and  conventions  based 
on  team  experience. 

•  Apply  mathematical  techniques 
in  state  and  clear  box  designs  to 
define  complex  behavior  with 
precision. 

•  Develop  a  design  review 
protocol  for  team  development 
reviews  based  on  team 
experience. 

•  Establish  libraries  of  reusable, 
certified  designs. 

•  Incorporate  advances  in 
formal  verification 
methods  into  local 
practices. 

•  Use  trace  tables  as 
necessary  to  support 
mental  proofs  of 
correctness. 

•  Document  written  proofs 
of  correctness  as  required 
for  critical  system 
functions. 

•  Develop  verification 
protocols  and  extended 
proof  rules  for 
application-,  language-, 
and  environment-specific 
semantics. 

•  Incorporate  advances  in 
scientific  software  certification 
methods  into  local  practices. 

•  Apply  experience  of 
prior  Cleanroom  projects 
and  customers  in  setting 
reliability  targets. 

•  Employ  usage  analysis  to 
validate  functional 
specifications  and  plan 
increment  content. 

•  Use  automated  tools  to 
generate  self-checking  test 
cases. 

•  Collect  customer  usage 
data  to  track  conformance  of 
usage  distributions  to  actual 
field  use. 

•  Apply  and  evaluate  multiple 
reliability  models  for  best 
prediction  of  system 
reliability  in  the  development 
environment. 

•  Use  the  full  rigor  of  statistical 
process  control  to  analyze 
team  performance. 

•  Compare  team  performance 
with  locally-defined  process 
control  standards  for 
performance. 

•  Use  error  classification 
schemes  to  improve  specific 

.  Cleanroom  practices  in 
specification,  design, 
verification,  and  testing. 

_J _ _ _ i - - 
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lowing  verification,  increments  are  delivered  to 
the  test  team  for  first  execution. 

In  an  introductory  implementation,  usage  testing 
based  on  external  system  behavior  replaces  cover¬ 
age  testing  based  on  design  internals.  Usage  infor¬ 
mation  is  collected  by  analyzing  functional  specifi¬ 
cations  and  surveying  prospective  users  (where 
users  may  be  people  or  other  programs).  Based  on 
this  information,  a  high-level  usage  profile  is  devel¬ 
oped,  including  nominal  scenarios  of  use,  as  well  as 
error  and  stress  situations.  A  usage  profile  can  be 
recorded  in  systematic  structures  such  as  hierar¬ 
chical  decision  trees  that  embody  possible  usage 
patterns  in  compact  form.  Next,  test  scenarios  are 
defined  based  on  the  usage  profile.  The  idea  is  that 
the  test  cases  represent  realistic  scenarios  of  user 
interaction,  including  both  correct  and  incorrect  us¬ 
age.  For  example,  if  particular  system  functions  are 
used  frequently  in  particular  patterns  with  occa¬ 
sional  user  mistakes,  this  usage  should  be  reflected 
in  the  test  suite.  At  this  stage,  the  usage  profile  may 
not  be  extremely  precise  or  detailed,  but  it  does 
contain  sufficient  information  for  the  test  team  to 
generate  realistic  test  cases. 

The  effectiveness  of  the  development  process  is 
measured  by  system  performance  in  testing  with 
respect  to  predetermined  quality  standards,  such 
as  failure  rates  and  severity  levels.  (More  precise 
statistical  measures,  such  as  MTTF  and  improve¬ 
ment  ratio,  are  introduced  in  the  full  implemen¬ 
tation.)  If  test  results  show  that  the  development 
process  is  not  meeting  quality  objectives,  testing 
ceases  and  the  code  is  removed  from  the  machine 
for  redevelopment  and  reverification  by  the  de¬ 
velopment  team. 

Process  improvement  is  a  fundamental  activity  in 
an  introductory  implementation.  The  idea  is  to 
shift  from  informal  discussions  of  lessons  learned 
to  a  systematic,  documented  improvement  pro¬ 
cess.  Baseline  measurements  of  fundamental 
project  characteristics,  such  as  quality,  produc¬ 
tivity,  and  cost,  provide  a  basis  for  assessing  pro¬ 
gress  and  making  improvements.  The  quality  re¬ 
sults  of  usage  testing  can  guide  changes  to  the 
development  process.  In  addition,  customer  sat¬ 
isfaction  with  Cleanroom-developed  systems  can 
highlight  process  areas  requiring  improvements. 

Full  implementation.  Introductory  Cleanroom  im¬ 
plementation  establishes  a  framework  for  matur¬ 
ing  the  process  t6  a  full  implementation.  As  sum¬ 


marized  in  the  second  row  of  Table  2,  full 
implementation  adds  rigor  to  practices  estab¬ 
lished  in  the  introductory  phase  through  formal 
methods  of  box  structure  specification  and  de¬ 
sign,  correctness  verification,  statistical  testing, 
and  reliability  certification.  For  a  Cleanroom  pro¬ 
ject  of  substantial  size  and  complexity,  a  team- 
of-teams  approach  can  be  applied,  whereby  the 
hierarchical  structure  of  the  system  under  devel¬ 
opment  forms  the  basis  for  organizing,  partition¬ 
ing,  and  allocating  work  among  a  corresponding 
hierarchy  of  small  teams. 

An  opportunity  exists  for  more  extensive  cus¬ 
tomer  interaction  in  a  full  Cleanroom  implemen¬ 
tation.  Customers  can  be  provided  with  education 
on  Cleanroom  practices  to  improve  the  effective¬ 
ness  of  functional  and  usage  specification  analysis 
and  review.  In  addition,  prototypes  and  accumu¬ 
lating  increments  can  be  provided  to  customers 
for  evaluation  and  feedback. 

Managers  and  team  leaders  can  leverage  Clean¬ 
room  experience  into  additional  flexibility  in  in¬ 
cremental  development  to  accommodate  chang¬ 
ing  requirements,  and  shortfalls  and  windfalls  in 
team  performance  within  remaining  schedule  and 
budget.  Increment  planning  can  emphasize  early 
development  of  useful  system  functionality  for 
customer  feedback  and  operational  use. 

In  specification  and  design,  prototyping  and  ex¬ 
perimentation  are  encouraged  to  clarify  and  val¬ 
idate  requirements,  and  to  understand  and  doc¬ 
ument  semantics  of  interfacing  software.  The 
formal  syntax  and  semantics  of  box  structures  are 
used  for  black,  state,  and  clear  box  refinements. 
Black  boxes  and  state  boxes  are  recorded  in  an 
outer  syntax  of  formal  structures,  such  as  tran¬ 
sition  tables,  with  inner  syntax  expressed  in  pre¬ 
cise  conditional  rules ,  often  given  as  conditional 
concurrent  assignments  combined  with  precise 
natural  language.  In  clear  box  design,  intended 
functions  are  recorded  at  every  level  of  refine¬ 
ment,  expressed  in  conditional  concurrent  assign¬ 
ments  and  precise  natural  language. 

A  box-structured  system  is  specified  and  de¬ 
signed  as  a  hierarchy  of  boxes,  such  that  appro¬ 
priate  system  data  are  encapsulated  in  boxes,  pro¬ 
cessing  is  defined  by  using  box  services,  and 
every  use  of  a  box  service  occupies  a  distinct 
place  in  the  hierarchy.  Box  structures  promote 
early  identification  of  common  services,  that  is. 
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reusable  objects,  that  can  simplify  development 
and  improve  productivity.  Duplication  of  effort  is 
avoided  when  team  members  have  an  early 
awareness  of  opportunities  for  use  and  reuse  of 
common  services.  Rigorous  team  verification  re¬ 
views  are  conducted  for  all  program  structures, 
using  mental  proofs  of  correctness  based  on  box 
structure  theory  and  the  correctness  conditions  of 
the  Correctness  Theorem. 

Statistical  testing  involves  a  more  complete  and 
experimentally  valid  approach  than  in  an  intro¬ 
ductory  implementation.  Reliability  objectives 
are  established  and  extensive  analysis  of  antici¬ 
pated  system  usage  is  carried  out.  Comprehen¬ 
sive  specifications  of  the  population  of  possible 
system  inputs  are  defined  in  usage  probability  dis¬ 
tributions  recorded  in  formal  grammars  or  state 
transition  matrices.  Automated  tools  are  used  to 
randomly  generate  test  cases  from  the  distribu¬ 
tions,  and  the  correct  output  for  each  test  input  is 
defined  based  on  the  system  specification.  For 
example,  the  IBM  Cleanroom  Certification  Assis¬ 
tant  (CCA)27  automates  elements  of  the  statistical 
testing  process  based  on  a  formal  grammar  model 
for  usage  probability  distributions.  It  contains  a 
Statistical  Testcase  Generation  Facility  for  com¬ 
piling  distributions  (expressed  in  a  Usage  Distri¬ 
bution  Language)  and  creating  randomized  test 
cases.  Reliability  models  are  employed  to  mea¬ 
sure  system  reliability  based  on  test  results,  and 
the  development  process  for  each  increment  is 
evaluated  based  on  the  extent  to  which  reliability 
results  meet  objectives.  The  CCA  provides  an  au¬ 
tomated  reliability  model,  the  Cleanroom  Certi¬ 
fication  Model,  that  analyzes  test  results  to  com¬ 
pute  mttf,  improvement  ratio,  and  other  sta¬ 
tistical  measures.  Alternative  distributions  are 
often  employed  to  certify  the  reliability  of  special 
aspects  of  system  behavior,  for  example,  infre¬ 
quently  used  functions  that  exhibit  high  conse¬ 
quences  of  failure. 

Process  improvement  is  established  through  re¬ 
views,  following  completion  of  each  increment, 
to  incorporate  team  recommendations  into  the 
documented  Cleanroom  process.  Causal  analysis 
of  failures  and  comprehensive  customer  surveys 
can  provide  additional  insight  into  process  areas 
requiring  improvement. 

Advanced  implementation.  Key  elements  of  an  ad¬ 
vanced  implementation  are  summarized  in  the 
third  row  of  Table  2.  At  this  level  of  experience. 
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the  Cleanroom  process  is  optimized  for  the  local 
environment  and  continually  improved  through 
advances  in  the  software  engineering  technology. 

A  Cleanroom  center  of  competency  can  be  es¬ 
tablished,  staffed  by  expert  practitioners  to  mon¬ 
itor  advances  in  Cleanroom  technology  and  pro¬ 
vide  training  and  consultation  to  project  teams. 

The  Cleanroom  process  can  be  scaled  up  to  ever 
larger  projects  and  applied  across  an  organiza¬ 
tion.  An  opportunity  exists  to  achieve  Cleanroom 
quality,  productivity,  and  cost  improvements 
even  in  emergency  and  adverse  system  develop¬ 
ments. 

Product  warranties  may  be  possible  in  customer 
contracts,  based  on  certification  with  usage  dis¬ 
tributions  and  reliability  models  agreed  to  by  both 
parties.  In  the  future,  a  capability  for  developing 
software  with  warranted  reliability  could  become 
a  major  differentiating  characteristic  of  software 
development  organizations.  Customers  can  ben¬ 
efit  by  capturing  actual  usage  from  specially  in¬ 
strumented  versions  of  Cleanroom-developed 
systems,  to  permit  test  teams  to  improve  the  ac¬ 
curacy  of  usage  distributions  employed  in  certi¬ 
fication. 

Incremental  development  can  be  used  to  manage 
project  risk  through  early  development  of  key  in¬ 
terfaces  with  pre-existing  software,  important 
user  functions,  and  performance-sensitive  com¬ 
ponents.  Increments  can  also  be  defined  to  isolate 
and  reduce  dependence  on  areas  of  incomplete  or 
volatile  requirements,  and  to  focus  on  early  ini¬ 
tiation  of  complex,  long-lead-time  components. 
Advanced  incremental  development  also  in¬ 
cludes  systematic  reuse  and  reliability  planning, 28 
facilitated  by  such  tools  as  the  Cleanroom  Reli¬ 
ability  Manager.29  In  this  approach,  libraries  of 
reusable  components  are  searched  for  functions 
identified  in  specification  and  top-level  design.  If 
the  reliability  of  candidate  components  is  not 
known,  statistically  valid  experiments  are  con¬ 
ducted  to  estimate  reliability.  If  reliability  of  a 
candidate  component  has  previously  been  certi¬ 
fied,  the  usage  profile  used  in  that  certification  is 
compared  with  the  new  usage  profile  to  determine 
if  the  previous  certification  is  valid  for  the  new 
use.  Once  reliability  estimates  exist  for  new  and 
reused  components,  an  estimate  of  total  system 
reliability  is  generated  through  calculations  based 
on  top-level  transition  probabilities  between  sub¬ 
systems.  The  results  of  this  analysis  are  used  to 
set  reliability  requirements  for  components,  eval-  • 
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uate  the  viability  of  component  reuse,  and  factor 
reliability  risks  into  increment  planning. 

An  advanced  use  of  box  structure  specification  in¬ 
volves  formal  mathematical  and  computer  science 
models  appropriate  to  the  application.  Formal  black 
box  and  state  box  outer  syntax  used  in  full  Clean- 
room  implementation  is  combined  with  formal  in¬ 
ner  syntax  expressed  as  propositional  logic,  pred¬ 
icate  calculus,  algebraic  function  composition,  BNF 
(Backus  Naur  form)  grammars,  or  other  formal  no¬ 
tation  that  affords  a  clear  and  concise  representa¬ 
tion  of  function.  Clear  box  designs  are  expressed  in 
design  languages  for  which  target  language  code 
generators  exist,  or  in  restricted  subsets  of  imple¬ 
mentation  languages,  thereby  eliminating  opportu¬ 
nities  for  new  errors  in  translation. 

In  verification  reviews,  trace  tables  are  employed 
where  appropriate  for  analysis  of  correctness, 
and  written  proofs  are  recorded  for  critical  func¬ 
tions,  particularly  in  life-,  mission-,  and  enter¬ 
prise-critical  systems.  Application-,  language-, 
and  environment-specific  proof  rules  and  stan¬ 
dards  provide  a  more  complete  framework  for 
team  verification.  Locally-defined  standards  have 
been  shown  to  be  more  effective  than  generic 
standards  in  producing  consistent  practitioner 
judgment  about  software  quality.30  In  an  ad¬ 
vanced  implementation,  the  documented  process 
includes  environment-specific  protocols  for  spec¬ 
ification,  design,  and  verification  based  on  team 
experience. 


An  advanced  implementation  can  benefit  from  a 
locally-validated  reliability  model  for  software 
certification.  Just  as  locally-validated  standards 
enable  more  consistent  practitioner  judgment 
about  software  quality,  a  locally-validated  reli¬ 
ability  model  will  enable  more  accurate  predic¬ 
tion  of  operational  reliability  from  testing  results. 

In  an  advanced  implementation,  the  full  rigor  of 
statistical  process  control  can  be  applied  to  pro¬ 
cess  improvement.  Team  accomplishments  can 
be  compared  to  locally-defined  process  control 
standards  for  performance.  Errors  can  be  cate¬ 
gorized  according  to  an  error  classification 
scheme  to  target  specific  Cleanroom  practices  for 
improvement. 

Choosing  an  implementation  approach 

Cleanroom  software  engineering  represents  a 
shift  from  a  paradigm  of  traditional,  craft-based 
practices  to  rigorous,  engineering-based  prac¬ 
tices,  specifically  as  follow. 


From: 

Individual  operations  -> 

Waterfall  development  — ► 

Informal  specification  -» 

Informal  design  -*> 

Defect  correction  -*■ 

Individual  unit  testing  -> 

Path-based  inspection  -> 

Coverage  testing  -> 

Indeterminate  reliability 


To: 

Team  operations 
Incremental  development 
Black  box  specification 
Box  structure  refinement 
Defect  prevention 
Team  correctness  verification 
Function-based  verification 
Statistical  usage  testing 
Certified  reliability 


In  an  advanced  approach  to  statistical  testing, 
Markov-  or  grammar-based  automated  tools  can 
be  used  to  improve  efficiency  and  effectiveness. 
For  example,  the  IBM  Cleanroom  Certification 
Assistant  permits  generation  of  any  required 
number  of  unique,  self-checking  test  cases.  In  ad¬ 
dition,  the  rich  body  of  theory,  analytical  results, 
and  computational  algorithms  associated  with 
Markov  processes  have  important  applications  in 
software  development.31  Both  formal  grammar 
and  Markov  usage  models  can  reveal  errors,  in¬ 
consistencies,  ambiguities,  and  data  dependen¬ 
cies  in  specifications  early  in  development,  and 
serve  as  test  case  generators  for  statistical  testing. 
Initial  versions  of  systems  can  be  instrumented  to 
record  their  own  usage  on  command,  as  a  base¬ 
line  for  analysis  and  calibration  of  usage  distri¬ 
butions  in  certification  of  subsequent  system  ver¬ 
sions.  ^ 


A  phased  approach  to  Cleanroom  implementation 
enables  an  organization  to  build  confidence  and 
capability  through  gradual  introduction  of  new 
practices  with  corresponding  growth  in  process 
control.  If  organizational  support  and  capability 
is  sufficient  for  full  implementation,  the  highest 
software  quality  and  reliability  afforded  by  Clean¬ 
room  practices  can  be  achieved.  Otherwise,  a 
phased  implementation  is  recommended.  In  gen¬ 
eral,  a  software  organization  that  employs  infor¬ 
mal  methods  of  specification  and  design,  relies  on 
coverage  testing  and  defect  correction  to  achieve 
quality,  and  has  little  experience  with  team-based 
operations,  can  gain  the  most  benefit  through  an 
introductory  implementation.  This  first  phase  in¬ 
troduces  a  comprehensive  set  of  practices  span¬ 
ning  project  management,  development,  and  test¬ 
ing,  but  without  the  full  formality  of  Cleanroom 
technology.  Once  an  organization  successfully 
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completes  a  project  using  the  introductory  prac¬ 
tices,  it  has  prepared  itself  for  a  full  implementa¬ 
tion.  Likewise,  maturation  from  full  to  advanced 
implementation  can  occur  when  the  practices  of 
the  second  stage  have  been  successfully  demon¬ 
strated. 

Note  that  very  few  teams  in  reality  will  implement 
the  precise  set  of  practices  defined  within  each 
implementation.  Each  team  embodies  unique 
skills,  processes,  and  experiences  that  must  be 
assessed  when  choosing  an  appropriate  imple¬ 
mentation.  It  is  often  the  case  that  a  team  can  best 
utilize  practices  from  more  than  one  implemen¬ 
tation  level.  For  example,  a  team  using  an  intro¬ 
ductory  implementation  may  have  had  prior 
experience  with  inspections  and  code  reviews. 
Consequently,  it  may  shift  to  a  full  or  advanced 
implementation  of  the  system  design  and  verifi¬ 
cation  practices.  Perhaps  another  mature  Clean- 
room  team,  using  primarily  advanced  practices, 
will  find  the  rigor  of  the  second  phase  of  system 
specification  to  be  sufficient. 

The  well-known  Software  Engineering  Institute 
Capability  Maturity  Model  provides  a  useful  as¬ 
sessment  technique  to  help  define  the  best 
Cleanroom  approach.32-33  In  general,  higher  as¬ 
sessment  levels  indicate  that  an  organization  can 
successfully  adopt  a  more  complete  Cleanroom 
implementation.  Organizations  assessed  at  levels 
1  and  2  will  likely  benefit  from  an  introductory 
implementation,  at  levels  2  and  3,  a  full  imple¬ 
mentation,  and  at  levels  4  and  5,  an  advanced 
implementation. 

Phased  implementation  on  the 
AOEXPERT/MVS  project 

AOEXPERT/MVS  is  the  largest  completed  Clean¬ 
room  project  in  IBM,  both  in  terms  of  lines  of  code 
and  project  staffing.  The  project  adopted  an  in¬ 
troductory  implementation  of  the  Cleanroom  pro¬ 
cess  for  development,  and  realized  a  defect  rate 
of  2.6  errors/KLOC,  measured  from  the  first  exe¬ 
cution  of  the  code.  This  represents  all  errors  ever 
found  in  testing  and  installation  at  three  field  test 
sites.  Development  productivity  averaged  486 
lines  of  code  per  person-month,  including  all  de¬ 
velopment  labor  expended  in  specification,  de¬ 
sign,  and  testing.  In  short,  the  AOEXPERT/MVS 
team  produced  a  complex  systems  software  prod¬ 
uct  with  an  extraordinarily  low  error  rate,  while 
maintaining  high  productivity.  The  following 


summary  of  the  project  is  elaborated  in  Reference 
15. 

The  AOEXPERT/MVS  product.  AOEXPERT/MVS  is 
a  decision-support  facility  that  uses  artificial 


Few  teams  will  implement 
the  precise  set  of 
practices  defined  within 
each  implementation. 


intelligence  (ai)  for  predicting  and  preventing  com¬ 
plex  operating  problems  in  an  MVS  environment. 
Primarily  a  host-based  product,  it  runs  in  a 
Net  View*  environment  on  MVS  with  interfaces  to 
several  other  IBM  program  products.  A  workstation 
component  running  under  Operating  System/2* 
(OS/2*)  in  the  Personal  System/2*  (PS/2*)  environ¬ 
ment  provides  the  user  interface  for  the  definition 
and  management  of  the  business  policies  for  system 
operation  to  be  applied  by  AOEXPERT/MVS  to  avoid 
and  correct  system  problems. 

The  complex  development  environment  required 
expertise  in  MVS  and  its  subsystems,  expert  sys¬ 
tems  technology,  real-time  tasking,  message 
passing,  and  windows-based  programming  for  the 
workstation  component.  The  product  was  imple¬ 
mented  using  PL/l,  TIRS*  (an  ai  shell),  PL /x  (an 
internal  IBM  system  language),  assembler,  JCL, 
and  REXX  for  host  software,  and  C  and  Presen¬ 
tation  Manager*  for  workstation  software.  The 
environment  was  further  complicated  by  two  ma¬ 
jor  dependencies  on  IBM  system  management 
products  that  were  developed  by  other  IBM  lab¬ 
oratories. 

The  project  began  in  July  1989,  with  the  first  eigh¬ 
teen  months  spent  in  the  requirements  phase.  De¬ 
velopment  team  staffing  took  place  during  this  ini¬ 
tial  stage.  Four  departments  were  ultimately 
established:  one  for  requirements,  two  for  devel¬ 
opment,  and  one  for  testing.  Various  support  orga¬ 
nizations  provided  market  development,  quality 
assurance,  information  development,  usability 
analysis,  and  business  and  legal  services. 
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Table  3  The  AOEXPERT/MVS  implementation  of  the 
Cleanroom  process 


Cleanroom  Practice  Introductory  Full  Advanced 

Team  operations 

X 

Customer  interaction 

X 

Incremental  development 

X 

System  specification 

X 

System  design 

X 

Correctness  verification 

X 

Statistical  usage  testing 

X 

Reliability  certification 

X 

Process  improvement 

X 

The  project  team  was  newly  formed,  with  mem¬ 
bers  ranging  from  programmer  retrainees  to  sen¬ 
ior  programmers  with  25  years  of  development 
experience.  The  project  team  averaged  50  people 
throughout  development.  Experience  in  the  prod¬ 
uct  domain  was  mixed,  with  considerable  expe¬ 
rience  in  application  development  and  AI,  but 
very  little  in  MVS  and  system  programming.  As  it 
turned  out,  AI  skills  were  utilized  about  10  percent 
of  the  time  during  development,  while  MVS  and 
system  programming  skills  were  needed  90  per¬ 
cent  of  the  time. 

This  was  the  first  Cleanroom  development  experi¬ 
ence  for  all  participants,  with  the  exception  of  one 
development  manager  and  two  developers.  Conse¬ 
quently,  extensive  education  and  training  were  re¬ 
quired  to  implement  Cleanroom  practices.  The 
overall  project  schedule  had  been  established  in  late 
1989,  prior  to  the  decision  to  use  the  Cleanroom 
process.  Given  the  schedule  and  mix  of  skills  and 
experience  levels,  the  Cleanroom  process  was  first 
met  with  healthy  skepticism.  The  team  had  to  grap¬ 
ple  with  three  important  factors  at  once:  a  new 
team,  little  experience  in  the  subject  domain,  and 
the  new  Cleanroom  development  process. 

Defining  an  introductoiy  implementation.  The  deci¬ 
sion  to  use  the  Cleanroom  process  was  made  in  the 
second  quarter  of  1990,  a  year  after  the  project 
started  and  six  months  prior  to  the  beginning  of 
development.  Due  to  the  aggressive  project  sched¬ 
ule,  the  large  size  of  the  organization,  the  lack  of 
prior  Cleanroom  experience,  and  the  limited 
amount  of  training  time  available,  the  management 
and  technical  team  decided  on  a  phased  implemen¬ 
tation  of  the  Cleanroom  approach.  As  summarized 
in  Table  3,  the  team  defined  an  introductoiy  ap¬ 
proach  that  included  team-based  operations,  exter¬ 


nal  specification  of  behavior  using  intended  func¬ 
tions,  design  expressed  in  a  Process  Design 
Language  (pdl)  with  automatic  target  translation 
(for  PL/i),  and  staged  delivery  of  each  increment  to 
independent  testers  for  first  execution.  In  addition 
to  the  introductory  practices,  two  full  practices 
were  used:  incremental  development  and  team- 
based  correctness  verification  of  every  line  of  code. 
While  it  was  agreed  that  statistical  testing  would  be 
very  effective,  the  test  team  did  not  believe  it  could 
learn  and  apply  the  methodology  in  time  for  the  first 
increment.  The  greatest  concern  was  the  late  start 
on  defining  a  usage  probability  distribution,  a  task 
normally  initiated  as  soon  as  the  functional  speci¬ 
fication  is  available.  The  test  team  initially  followed 
the  spirit  if  not  the  form  of  usage  testing,  with  a 
testing  approach  based  on  expected  customer  us¬ 
age.  Later,  statistical  usage  testing  was  employed 
for  a  significant  subset  of  the  product,  the  worksta¬ 
tion  component,  which  accounted  for  approxi¬ 
mately  40  percent  of  total  product  code. 

Getting  started.  Cleanroom  education  was  pro¬ 
vided  to  the  entire  project,  with  mandatory  man¬ 
agement  participation.  To  further  define  the  use 
of  Cleanroom  process  in  the  project  environment, 
a  process  working  group  was  formed  to  document 
the  AOEXPERT/MVS  Cleanroom  development  pro¬ 
cess,  to  establish  and  maintain  project  proce¬ 
dures,  standards,  and  conventions,  to  establish 
and  maintain  a  measurement  and  improvement 
subprocess,  and  to  provide  a  formal  mechanism 
to  resolve  process  issues  and  make  improvements. 
Each  major  project  functional  area,  including  ar¬ 
chitecture,  host  development,  workstation  devel¬ 
opment,  test,  configuration  management,  and  qual¬ 
ity  assurance,  was  represented  on  the  process 
working  group.  The  group  documented  a  compre¬ 
hensive  set  of  procedures  and  standards  for  an  in¬ 
tegrated,  Cleanroom-based  software  development 
process.  This  document  and  its  subsequent  use  by 
the  team  was  critical  in  achieving  acceptance  and 
ownership  of  the  process  by  the  team.  Changes  to 
the  process  required  approval  by  the  process  work¬ 
ing  group  and  management.  During  the  develop¬ 
ment  of  aoexpert/mvs,  a  number  of  useful  process 
revisions  resulted  from  suggestions  by  team  mem¬ 
bers  in  periodic  meetings  held  to  improve  the  de¬ 
velopment  process. 

Applying  the  introductoiy  implementation.  The 

decision  to  use  the  Cleanroom  process  was  made 
rather  late  in  the  project  after  the  product  func- 
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tional  specification  (PFS)  document  was  almost 
completed.  The  PFS  is  required  for  IBM  program 
product  development,  but  it  is  not  an  adequate 
replacement  for  a  Cleanroom  specification,  as  it 
contains  only  a  subset  of  the  information  re¬ 
quired.  The  AOEXPERT/MVS  team  decided  to  com¬ 
plete  the  PFS,  and  then  produce  a  more  formal 
black  box,  incremental  specification.  The  formal 
specification  used  precise  English  descriptions  in 
conjunction  with  intended  functions  to  specify 
the  external  behavior  of  the  increments. 

Following  specification,  project  technical  leaders 
created  an  incremental  development  plan  that  de¬ 
fined  the  functional  content,  development  sched¬ 
ule,  and  resource  requirements  for  three  software 
increments.  Although  the  project  completion  date 
had  been  established  earlier,  substantial  flexibil¬ 
ity  remained  for  scheduling  increment  develop¬ 
ment  and  testing  within  the  overall  schedule  of  12 
months.  Historical  productivity  and  defect  rates 
from  comparable  traditionally-developed  applica¬ 
tions  were  reviewed  and  the  schedules  were  ad¬ 
justed  based  on  historical  Cleanroom  data,  personal 
experience,  and  confidence.  The  first  increment 
was  planned  to  contain  the  least  function  of  the 
three,  in  order  to  quickly  familiarize  the  project 
team  with  the  new  Cleanroom  process  and  devel¬ 
opment  environment.  Development  of  the  first  in¬ 
crement  required  two  and  one-half  months,  with  the 
second  and  third  increments  requiring  three  and 
one-half  months  each. 

Eight  principal  functional  components  were  de¬ 
fined  for  AOEXPERT/MVS  and  organized  into  func¬ 
tional  content  comprising  the  three  increments. 
Each  component  was  assigned  to  a  team  com¬ 
posed  of  from  one  to  five  developers,  with  each 
team  augmented  by  an  architect  and  a  tester. 
Team  membership  remained  stable  throughout 
development  of  all  three  increments,  helping  to 
ensure  continuity  and  growth  of  expertise  and  ca¬ 
pability.  A  functional  management  approach  was 
adopted  because  each  team  consisted  of  people 
from  different  departments.  Since  each  team  had 
a  designated  team  leader,  management  ownership 
was  assigned  based  on  the  team  leader.  Thus,  a 
manager  was  responsible  for  all  teams  led  by 
members  of  the  manager’s  department.  This  pro¬ 
cess  worked  well,  but  required  daily  communi¬ 
cation  between  managers,  usually  in  the  form  of 
morning  status  meetings  where  schedules,  plans, 
resources,  and  performance  were  addressed. 
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Following  increment  planning,  development  began 
for  the  first  increment.  It  immediately  became  ob¬ 
vious  that  the  developers  lacked  a  good  understand¬ 
ing  of  the  entry  criteria  for  team  correctness  veri¬ 
fication  reviews.  Most  understood  how  to  perform 
verification,  but  underestimated  the  level  of  rigor 
and  precision  required  in  the  design  material.  For 
example,  intended  functions  documented  in  many 
of  the  early  first  increment  designs  precisely  spec¬ 
ified  intended  behavior  for  normal  or  steady-state 
operation,  but  failed  to  specify  intended  behavior 
for  error  conditions,  exception  processing,  and  un¬ 
expected  input.  As  a  result,  the  designs  could  not  be 
verified  for  correctness. 

To  address  this  problem,  project  management  de¬ 
cided  that  a  demonstration  verification  review  of 
an  actual  first  increment  design  should  be  held  as 
early  as  possible.  A  senior-level  programmer  was 
asked  to  prepare  a  design  for  the  review.  When 
the  design  was  ready,  his  five-member  team  con¬ 
ducted  a  formal  correctness  verification  review, 
with  the  remainder  of  the  AOEXPERT/MVS  organi¬ 
zation,  numbering  about  45  people,  in  attendance 
as  observers.  Everyone  in  attendance  had  a  copy 
of  the  material  and  followed  along  with  the  review 
team.  The  review  lasted  about  three  hours,  with 
the  design  failing  to  pass  the  verification  process. 

This  outcome  proved  to  be  an  invaluable  teaching 
tool  for  the  project  team.  Most  were  surprised 
that  the  design  did  not  pass,  and  even  more  sur¬ 
prised  at  the  number  of  changes  required  to  make 
it  verifiable.  The  demonstration  clearly  showed 
the  team  what  was  actually  expected  in  a  Clean¬ 
room  review,  and  definitely  saved  a  substantial 
amount  of  time  and  frustration  in  the  remainder  of 
the  project.  Since  the  first  increment  was  rela¬ 
tively  small  and  straightforward,  the  team  was 
able  to  learn  how  to  correctly  apply  the  Clean¬ 
room  approach  and  still  make  the  first  delivery 
date. 

Cleanroom  facilitators.  The  AOEXPERT/MVS  proj¬ 
ect  benefited  from  people  with  prior  Cleanroom 
experience,  who  played  dual  roles  as  team  mem¬ 
bers  and  Cleanroom  methodology  consultants. 
These  people  served  as  teachers  and  advisors, 
providing  guidance  on  how  to  write  verifiable  de¬ 
signs  and  conduct  effective  verification  reviews. 
Equally  important  was  the  encouragement  they 
gave  and  confidence  they  instilled  in  their  peers 
through  their  example  and  coaching.  During  the 
first  increment  of  development,  one  of  these  ex- 
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Table  4  AOEXPERT/MVS  error  rates  measured  from 
first  execution 

AOEXPERT/MVS  Industry  AOEXPERT/MVS 


Project _ Expectation  Project  Results 


Incre¬ 

ment 

KLOC 

Errors  Projected 
at  30/  Errors 
KLOC 

Actual 

Software 

Errors 

Errors/ 

KLOC 

1 

16 

480 

64 

43 

2.7 

2 

50 

1500 

200 

41 

0.8 

3 

41 

1230 

164 

97 

2.4 

Subtotal 

107 

3210 

428 

181 

L7 

System 

testing 

107 

93 

0.9 

Total 

107 

3210 

53 5 

274 

Z6 

Where 


•  Projected  errors  included  increment  testing  projected 
at  4  errors/KLOC,  and  system  testing  at  1  error/KLOC 

•  Actual  software  errors  were  measured  from  the  first 
execution 

•  System  testing  included  system,  performance,  and 
field  testing 


perts  was  present  at  every  verification  review  to 
ensure  the  methodology  was  followed,  especially 
with  respect  to  application  of  the  correctness  ver¬ 
ification  conditions.  During  development  of  the 
second  and  third  increments,  other  team  mem¬ 
bers,  now  with  experience  in  the  Cleanroom  pro¬ 
cess,  joined  with  the  original  experts  to  form  a 
core  group  of  five  to  six  facilitators  who  served  a 
key  role  in  acceptance,  application,  and  improve¬ 
ment  of  the  Cleanroom  process. 

Team  verification  reviews.  The  Cleanroom  cor¬ 
rectness  verification  process  was  closely  fol¬ 
lowed.  A  check  was  made  prior  to  every  review 
to  ensure  that  the  entry  criteria  were  satisfied, 
and  a  disciplined  process  of  correctness  condition 
verification  for  every  control  structure  was  fol¬ 
lowed  during  the  review  process.  A  moderator 
was  assigned,  usually  one  of  the  Cleanroom  fa¬ 
cilitators,  to  ensure  that  the  reviews  were  con¬ 
ducted  properly,  and  that  all  issues  were  recorded 
and  all  changes  reverified.  The  author  of  the  de¬ 
sign  under  verification  typically  led  the  team 
through  the  review.  Also  present  were  a  key  re¬ 
viewer,  usually  the, component  team  leader  who 
had  a  broad  understanding  of  the  component 
function,  and  other  reviewers  j  typically  members 


of  other  teams  whose  components  interfaced  with 
the  designs  under  review.  Review  materials  were 
required  to  be  distributed  to  all  reviewers  at  least 
48  hours  prior  to  the  review,  and  all  reviewers 
were  expected  to  have  read  the  materials  before 
attending  the  review. 

Quality  results.  The  AOEXPERT/MVS  testing  pro¬ 
cess  was  composed  of  two  phases,  increment 
testing  and  system  testing.  (In  a  full  implementa¬ 
tion  of  the  Cleanroom  process,  all  testing  would 
be  regarded  as  system  testing.)  After  examining 
data  from  prior  Cleanroom  projects,  the  test  team 
estimated  expected  defect  rates  in  testing  and 
customer  use  of  the  product.  Four  errors/KLOC 
were  estimated  for  increment  testing,  an  addi¬ 
tional  1  error/KLOC  for  system  testing,  and  an  ad¬ 
ditional  0.5  error/KLOC  for  customer  use  after  the 
product  was  shipped.  These  estimates  were  sig¬ 
nificantly  lower  than  those  customarily  found  for 
comparable  products,  but  the  team  believed  that 
such  aggressive  goals  should  be  set,  even  for  a 
first-time  Cleanroom  effort. 

Table  4  summarizes  error  rates  for  the  three  prod¬ 
uct  increments,  measured  from  the  first  execution 
of  the  code.  For  comparison,  projected  errors 
are  shown  based  on  an  average  industrial  rate  of 
30  errors/KLOC20  for  traditional  development 
projects  measured  from  the  first  execution  of  the 
code,  with  a  total  of  3210  errors  expected  at  this 
rate.  The  test  team  estimate  of  5  errors/KLOC  (4  in 
increment  testing  plus  1  in  system  testing)  totaled 
to  535  errors  expected. 

The  aoexpert/mvs  team  produced  the  complex 
systems  software  product  with  only  274  errors 
found  in  all  testing.  This  error  rate  of  2.6 
errors/KLOC  was  over  an  order  of  magnitude  bet¬ 
ter  than  the  industry  average  of  30  errors/KLOC, 
and  nearly  halved  the  projected  Cleanroom  rate 
of  5  errors/KLOC.  A  number  of  system  compo¬ 
nents  completed  testing  with  no  errors  found.  For 
example,  five  of  the  eight  components  in  the  first 
16  KLOC  increment  proved  to  be  error-free  in  all 
testing.  In  addition,  no  operational  errors  what¬ 
soever  were  found  following  product  installation 
at  three  customer  test  sites,  and  no  post-ship  cus¬ 
tomer  errors  have  been  reported  to  date. 

Productivity  results.  Productivity  estimates  for 
AOEXPERT/MVS  were  based  on  rates  for  compa¬ 
rable,  traditionally-developed  products,  modified 
by  expected  gains  from  the  Cleanroom  process 
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and  the  belief  that  productivity  would  improve 
with  each  successive  increment.  Productivity 
was  estimated  at  300  lines  of  code  per  person- 
month  (LOC/PM)  for  the  first  increment,  350  for  the 
second  increment,  and  400  for  the  third  incre¬ 
ment.  Table  5  shows  actual  productivity  rates 
achieved,  based  on  total  lines  of  code  divided  by 
the  person-months  accumulated  for  formal  spec¬ 
ification  through  testing  of  the  final  increment. 
The  person-months  include  development  staff 
only.  The  project  achieved  very  competitive  pro¬ 
ductivity  rates,  exceeding  the  projected  rates  by 
36  percent  overall.  This  substantial  improvement 
in  productivity  was  a  significant  factor  in  enabling 
the  project  to  meet  its  schedule.  The  original  code 
size  estimate  was  72  KLOC,  but  the  actual  code 
size  was  significantly  larger  (107  KLOC)  due  pri¬ 
marily  to  unexpected  growth  in  the  workstation 
software  (from  10  to  42  KLOC).  The  growth  re¬ 
sulted  from  the  lack  of  familiarity  with  OS/2  Pre¬ 
sentation  Manager  and  unanticipated  require¬ 
ments.  Thus,  while  actual  productivity  was  a  36 
percent  improvement  over  the  projected  rate,  ac¬ 
tual  code  size  was  49  percent  larger  than  planned. 
The  increased  productivity  enabled  the  team  to 
stay  on  schedule  during  the  development. 

Observations.  From  the  beginning  of  the  project 
through  delivery  and  testing  of  the  first  incre¬ 
ment,  many  developers  and  testers  were  some¬ 
what  skeptical  about  the  Cleanroom  approach. 
The  real  turnaround  in  acceptance  occurred  after 
the  first  increment  was  delivered  and  tested  and 
so  few  errors  were  found.  In  fact,  several  testers 
were  upset  and  worried  when  they  failed  to  find 
any  errors;  ironically,  so  were  the  developers. 
But  this  soon  changed  for  everyone— defects 
quickly  became  the  exception,  not  the  rule,  and  a 
“right  the  first  time”  psychology  took  hold. 

The  challenges  facing  a  new  team  in  an  unfamiliar 
environment  were  great,  and  schedules  and  re¬ 
sources  were  extremely  tight.  Nevertheless,  a 
new  methodology  was  introduced,  taught,  and 
implemented  with  substantial  success.  The  pri¬ 
mary  success  factors  in  this  implementation  of 
Cleanroom  process  were  the  use  of  an  introduc¬ 
tory  implementation,  early  and  ongoing  manage¬ 
ment  commitment,  incremental  development  of 
system  function,  demonstration  reviews  for  team 
education,  team-based  peer  review  of  all  work 
products,  full  application  of  correctness  verifica¬ 
tion,  adherence  to  defect  prevention  practices,  and 
the  use  of  Cleanroom  consultants  and  facilitators. 


Table  5  AOEXPERT/MVS  productivity  rates 


Incre- 

KLOC 

Projected 

Actual 

%  Actual 

merit 

Productivity 

Productivity 

Exceeds 

LOC/PM 

LOC/PM 

Projected 

1 

16 

300 

400 

+33 

2 

50 

350 

500 

+43 

3 

41 

400 

513 

+28 

Average 

358 

486 

+36 

Where  the  actual  productivity 

was  the  LOC/PM 

measured  from  formal  specification  through  testing  | 

The  AOEXPERT/MVS  experience  is  representative 
of  the  new  level  of  quality  that  is  possible  in  soft¬ 
ware  development  today.  Cleanroom  is  a  practi¬ 
cal  and  proven  alternative  to  the  high  cost  and 
poor  quality  frequently  seen  in  traditional  devel¬ 
opment  processes.  As  evidence  of  its  effective¬ 
ness  continues  to  accumulate,  the  Cleanroom 
process  will  be  increasingly  adopted  by  organi¬ 
zations  seeking  competitive  business  advantage. 
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The  philosophy  behind 
Cleonroom  software 
engineering  is  to  avoid 
dependence  on  costly 
defect-removal  processes 
by  writing  code  increments 
right  the  first  time 
and  verifying  their 
correctness  before  testing. 

Its  process  model 
incorporates  the  statistical 
quality  certification  of  code 
increments  as  they 
accumulate  into  a  system. 

RICHARD  C.  LINGER 
IBM  Cleomoom  Software 
Technology  Center 


CLEANROOM 
PROCESS  MODEL 


Today’s  competitive  pressures  and  : 

society’s  increasing  dependence  on 
software  have  led  to  a  new  focus  on  devel¬ 
opment  processes.  The  Cleanroom  pro¬ 
cess,  which  has  evolved  over  the  last  de¬ 
cade,  has  demonstrated  that  it  can  improve 
bo di  die  productivity  of  developers  who 
use  it  and  die  quality  of  the  software  they 
produce. 

Cleanroom  software  engineering  is  a 
team-oriented  process  diat  makes  devel¬ 
opment  more  manageable  and  predictable 


This  article  is  based  on  a  pa|XT  that  apjx-ars  in  hvc.  1 5th 
Int'l  Conf.  Sojh'rtrv  Eng.,  IPKE  GS  Press,  I-os  Alamitos, 
Calif.,  1993, pp.  2- 13.  'Hie lEEESojhwv Editorial  Board 
has  selected  it  as  the  best  practice  paper  presented  at 
ICSP.-15/Ihe  Ixmrd  thanks  Richard  A.  De.Millo,  ICSP- 
15  Pn>«rJtn  G>chair,  for  his  help  in  arranging  for  its  pub¬ 
lication  in  lEEESofkitinr. 


\  because  it  is  done  under  statistical  quality 
control. 

Cleanroom  is  a  modem  approach  to 
software  development.  In  traditional, 
craft-based  development,  defects  are  re¬ 
garded  as  inevitable  and  elaborate  defect- 
removal  techniques  are  a  part  of  the  devel¬ 
opment  process.  In  such  a  process, 
software  proceeds  from  development  to 
unit  testing  and  debugging,  then  to  func¬ 
tion  and  system  testing  for  more  debug¬ 
ging.  In  the  absence  of  workable  alterna¬ 
tives,  managers  encourage  programmers 
to  get  code  into  execution  quickly,  so  de¬ 
bugging  can  begin.  Today,  developers  rec¬ 
ognize  that  defect  remoral  is  an  error- 
prone,  inefficient  activity  that  consumes 
resources  better  allocated  to  getting  the 
code  right  the  first  time. 
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Clean  room  teams  ar  IBM  and  other 
organizations  are  achieving  remarkable 
qua  lit}’  results  in  both  new- system  devel¬ 
opment  and  modifications  and  extensions 
to  legacy  systems.  The  quality  of  software 
produced  by  Clennroom  development 
teams  is  sufficient  (often  near  zero  defects) 
for  die  software  to  enter  system  testing 
directly  for  first-ever  execution  by  test 
teams. 

The  theoretical  foundations  of  Clean- 
|  room  —  formal  specification  and  design, 
j  correctness  verification,  and  statistical 
testing  —  have  been  reduced  to  practice 
and  demonstrated  in  nearly  a  million  lines 
of  code.  Some  Clean  room  projects  are 
profiled  in  the  box  on  p.  56. 

QUALITY  COMPARISON 

Quality  comparisons  between  tradi¬ 
tional  methods  and  the  (-lean room  pro¬ 
cess  are  meaningful  when  measured  from 
first  execution.  Most  traditional  develop¬ 
ment  methods  begin  to  measure  errors  at 
function  testing  (or  later),  omitting  errors 
found  in  private  unit  testing.  A  traditional 
project  experiencing,  say,  five  errors  per 
thousand  lines  of  code  (KLOQ  in  func¬ 
tion  testing  may  have  encountered  25  or 
more  errors  per  KLOC  when  measured 
from  first  execution  in  unit  testing. 

At  entry  to  unit  testing,  traditional  soft¬ 
ware  typicallv  exhibits  25  to  35  or  more 
errors  per  KLOC.1  In  contrast,  the 
weighted  average  of  errors  found  in  17 
Cleanroom  projects,  involving  nearly  a 
million  lines  of  code,  is  2.3  errors  per 
KLOC.  This  number  represents  all  errors 
found  in  all  testing,  measured  from  first- 
ever  execution  through  test  completion  — 
it  is  the  average  number  of  residual  errors 
present  after  the  development  team  has 
performed  correctness  verification. 

In  addition  to  this  remarkable  differ¬ 
ence  in  die  number  of  errors,  experience 
has  shown  a  qualitative  difference  in  die 
complexity  of  errors  found  in  Cleanroom 
versus  traditional  software.  Errors  left  be¬ 
hind  by  Cleanroom  correctness  verifica¬ 
tion  tend  not  to  be  complex  design  or  in¬ 
terface  errors,  but  simple  mistakes  easily 
found  and  fixed  by  statistical  testing. 

In  this  article,  I  describe  die  ('lean-  1 


room  development  process,  from  specifi¬ 
cation  and  design  through  correcuiess 
verification  and  statistical  usage  testing  for 
qua  1  i  ty  certi  fica  lion . 

INCREMENTAL  DEVELOPMENT 

The  Cleanroom  process  is  based  on 
developing  and  certifying  a  pipeline  of 
software  increments  that  accumulate  into 
the  final  system.  The  increments  are  de¬ 
veloped  and  certified  by  small,  indepen¬ 
dent  teams,  widi  teams  of  teams  for  large 
!  projects. 

System  integration  is  continual,  and 
functionality  grows  with 
;  the  addition  of  successive 
increments.  In  this  ap¬ 
proach,  the  harmonious 
operation  of  future  incre¬ 
ments  at  the  next  level  of 
refinement  is  predefined 
by  increments  already  in 
execution,  thereby  mini¬ 
mizing  interface  and  de¬ 
sign  errors  and  helping 
developers  maintain  in¬ 
tellectual  control.  i 

I'he  Cleanroom  de¬ 
velopment  process  is  in¬ 
tended  to  be  “quick  and  clean,”  not  “quick 
and  dirt}’.”  The  idea  is  to  quick!}'  develop 
die  right  product  widt  high  quality  for  die 
user,  then  go  on  to  die  next  version  to 
incorporate  new  requirements  arising 
from  user  experience.  j 

In  the  Cleanroom  process,  correctness 
is  built  in  by  the  development  team 
through  formal  specification,  design,  and 
verification.  Team  correctness  verification  j 
takes  die  place  of  unit  testing  and  debug¬ 
ging,  and  software  enters  system  testing 
directly,  with  no  execution  by  die  develop¬ 
ment  team.  All  errors  are  accounted  for 
from  first  execution  on,  with  no  private 
debugging  permitted. 

Figure  1  illustrates  the  Cleanroom  ' 
process  of  incremental  development  and  j 
quality  certification.  The  Cleanroom  j  i 
team  first  analyzes  and  clarifies  customer  ; 
requirements,  with  substantial  user  inter-  i  i 
action  and  feedback.  If  requirements  are  in  j  < 
doubt,  die  team  can  develop  Cleanroom  i 
prototypes  to  elicit  feedback  iteratively. 


(LEANROOM 
DEVELOPMENT 
IS  INTENDED 
TO  BE  "QUICK 
AND  CLEAN," 
NOT  "QUICK 
AND  DIRTY." 


As  die  figure  shows,  Cleanroom  devel¬ 
opment  involves  two  cooperating  teams 
and  five  major  activities: 

♦  Specification.  Cleanroom  develop¬ 
ment  begins  with  specification.  Together, 
die  development  team  and  die  certifica¬ 
tion  team  produce  two  specifications: 
functional  and  usage.  Large  projects  may 
have  a  separate  specification  team. 

The  functional  specification  defines 
the  required  external  system  behavior  in 
all  circumstances  of  use;  the  usage  specifi¬ 
cation  defines  usage  scenarios  and  their 
probabilities  for  all  possible  system  usage, 
bodi  correct  and  incorrect.  The  func¬ 
tional  specification  is  die 
basis  for  incremental  soft- 
100M  ware  development.  The 

iDMEIJT  usage  specification  is  die 

Till  till  basis  for  genera  tins  test 

^DED  cases  for  incremental  sta- 

tistical  testing  and  quality 
UUIU\  certification.  Usage  sped- 

EAN,  tt  fications  are  explained  in 

r  the  section  on  ce  rrifi  ea¬ 


rn  yy  //  ♦  Increment  planning. 

ml  i.  Op  the  basis  of  these 

sjiecifi cations,  die  devel-  ! 
opment  and  certification  ; 
teams  together  define  an  initial  plan  for  j 
developing  increments  that  will  accumu- 
late  into  the  final  system.  For  example,  a  j 
100  KLOC  system  might  be  develop'd  in  i 
five  increments  averaging  20  KLOC  each,  j 
The  time  it  takes  to  design  and  verify-  in-  j 
cremcncs  varies  with  their  size  and  com-  ■ 
plexity.  Increments  diar  require  long  lead  j 
times  may  call  for  parallel  development. 

♦  Design  and  verification.  The  d evelop - 
ment  team  dien  carries  out  a  design  and 
correctness  verification  cycle  for  each  in-  j 
crement.  The  certification  team  proceeds  ' 
in  parallel,  using  die  usage  specification  to  | 
generate  test  cases  that  reflect  the  ex-  j 
peered  use  of  the  accumulating  incre-  ! 
ments. 

♦  Quality  certification.  Periodically,  the 
development  team  integrates  a  completed 
increment  widi  prior  increments  anti  tie-  J 
livers  diem  to  die  test  team  for  execution  : 
of  statistical  test  cases.  The  test  cases  are  | 
run  against  the  accumulated  increments  . 
and  the  results  checked  for  correctness  : 
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Figure  L  Cleanroom  process  model  The  stacked  boxes  indicate  successive  increments. 


♦  Feedback.  Errors  are  returned  to  the 
development  team  for  correction.  If  the 
quality  is  low,  managers  and  team  mem¬ 
bers  initiate  process  improvement.  As  with 
any  process,  a  good  deal  of  iteration  and 
feedback  is  always  present  to  accommo¬ 
date  problems  and  solutions. 

hi  the  next  sections,  I  describe  the 
specification,  design  and  verification,  and 
quality-certification  procedures.  A  de¬ 
tailed  description  of  increment  planning 
and  feedback  mechanisms  is  outside  the 
scope  of  this  article. 

FUNCTIONAL  SPECIFICATION 

The  object- based  technology  of  box 
structures  has  proved  to  be  an  effective 
technique  for  functional  specification.3 
Through  stepwise  refinement,  objects  are 


against  the  functional  specification.  Inter¬ 
fail  times,  that  is,  die  elapsed  times  be¬ 
tween  failures,  are  passed  to  a  quality-cer¬ 
tification  model2  that  computes  objective 
statistical  measures  of  quality,  such  as 
mean  time  to  failure.  The  quality-certifi¬ 
cation  model  employs  a  reliability’  growth 
estimator  to  derive  the  statistical  mea¬ 
sures. 

Certification  is  done  continuously, 
over  the  life  of  the  project.  Higher  level 
increments  enter  the  certification  pipeline 
first.  This  means  major  architectural  and 
design  decisions  are  validated  in  execution 
before  the  development  team  elaborates 
on  them.  And  because  certification  is  done 
for  all  increments  as  they  accumulate, 
higher  level  increments  are  subjected  to 
more  testing  than  lower  level  increments, 
t  which  implement  localized  functions. 


defined  and  refined  as  different  box  struc¬ 
tures,  resulting  in  a  usage  hierarchy  of  ob¬ 
jects  in  which  the  services  of  an  object  may 
be  used  and  reused  in  many  places  and  at 
many  levels.  Box  structures,  dien,  define 
required  system  behavior  and  derive  and 
connect  objects  comprising  a  system  ar¬ 
chitecture.4-5 

In  die  past,  without  a  rigorous  specifi¬ 
cation  technolog}',  there  was  little  incen¬ 
tive  to  devote  much  effort  to  die  specifica¬ 
tion  process.  Specifications  were 
frequently  written  in  natural  language, 
with  inevitable  ambiguities  and  omissions, 
and  often  regarded  as  throwaway  stepping 
stones  to  code. 

Box  structures  provide  an  economic 
incentive  for  precision.  Initial  box-struc¬ 
ture  specifications  often  reveal  gaps  and 
misunderstandings  in  customer  require¬ 
ments  that  would  ordinarily  be  discovered 
later  in  development  at  high  cost  and  risk 
to  die  project. 

They  also  address  the  two  engineering 
problems  associated  widi  system  specifica¬ 
tion:  defining  the  right  function  for  users 
and  defining  the  right  structure  for  the  spec¬ 
ification  itself.  Box  structures  address  the 
first  problem  by  precisely  defining  the  cur¬ 
rent  understanding  of  required  functions  at 
each  stage  of  development,  so  that  the  func¬ 
tions  can  be  reviewed  and  modified  if  neces¬ 
sary.  The  second  problem  is  critical,  espe¬ 
cially  for  large-s}’stem  development.  How 
can  we  organize  the  myriad  details  of  be¬ 
havior  and  processing  into  coherent  ab¬ 
stractions  humans  can  understand? 

Box  structures  incorporate  the  crucial 
mathematical  property  of  referential 
transparency  —  the  information  content 
of  each  box  specification  is  sufficient  to 
define  its  refinement,  without  depending 
on  the  implementation  of  any  other  box. 
This  property  lets  us  organize  large-sys¬ 
tem  specifications  hierarchically,  without 
sacrificing  precision  at  high  levels  or  detail 
at  low  levels. 

Box  structures.  Three  principles  govern 
the  use  of  box  structures:4 

♦  All  data  defined  in  a  design  is  encap¬ 
sulated  in  boxes. 

♦  All  processing  is  defined  by  using 
boxes  sequentially  or  concurrently. 


[set  w  to 
minimum  of 
z  and  absolute 
value  of  x] 


[set  w  to 
minimum  of 
z  and  absolute 
value  of  x] 

DO 

[set  y  to  absolute 
value  of  x] 

[set  w  to  minimum 
of  z  and  y] 

END 


[set  w  to 
minimum  of 
z  and  absolute 
value  of  x] 

DO 

[set  y  to  absolute 
value  of  x] 

IF  x  <  0 
THEN 

y  :=  -x 

ELSE 

y  :=  x 

END 

[set  w  to  minimum 
of  z  and  y] 

IF  y  <  z 
THEN 
w  :  =  y 
ELSE 
w  :  =  z 
END 


Figure  2.  Stepwise  refinement  of a  dear-box  design  fragment  that  can  be  verified \  Each 
fragment  has  identical  functional  behavior ,  even  though  the  level  oj  detail  increases. 


♦  Each  box  occupies  a  distinct  place  in 
a  system’s  usage  hierarchy. 

Each  box  has  three  forms  —  black, 
state,  and  clear  —  which  have  identical 
external  behavior  but  whose  internals  are 
increasingly  detailed. 

Block  box.  An  object’s  black  box  is  a  pre¬ 
cise  specification  of  external,  user-visible 
behavior  in  all  possible  circumstances  of  its 
use.  The  object  may  be  an  entire  system  or 
any  part  of  a  system.  Its  user  may  be  a 
person  or  another  object. 

A  black  box  accepts  a  stimulus  (S)  from 
a  user  and  produces  a  response  (R).  Each 
response  of  a  black  box  is  determined  bv  its 
current  stimulus  history  (Sid),  with  a 
black-box  transition  function 

(S,  SH)  ->  (R) 

A  given  stimulus  will  produce  different 
responses  that  are  based  on  history  of  use, 
not  just  on  the  current  stimulus.  Imagine  a 
calculator  with  two  stimulus  histories 

Clear  7  1  3 

and 

Clear  713+ 

If  the  next  stimulus  is  6,  the  first  history 
produces  a  response  of  7136;  the  second, 

6. 

The  objective  of  a  black-box  specifica¬ 
tion  is  to  define  the  responses  produced 
for  every  possible  stimulus  and  stimulus 
history,  including  erroneous  and  unex¬ 
pected  stimuli.  By  defining  behavior  solely 
in  terms  of  stimulus  histories,  black-box 
specifications  neither  depend  on  nor  pre¬ 
maturely  define  design  internals. 

Black-box  specifications  are  often  re¬ 
corded  as  tables.  In  each  row,  the  stimulus 
and  the  condition  on  stimulus  history  are 
sufficient  to  define  the  required  response. 
To  record  large  specifications,  classes  of 
behavior  are  grouped  in  nested  tables  and 
compact  specification  functions  are  used 
to  encapsulate  conditions  on  stimulus  his¬ 
tories.6 

Slate  box .  An  object’s  state  box  is  derived 
from  its  black  box  by  identifying  the  ele¬ 
ments  of  stimulus  history  that  must  be  re¬ 
tained  as  state  data  between  transitions  to 
achieve  the  required  black-box  behavior. 

The  transition  function  of  a  state  box  is 

(S,  OS)  -»  (?.,  MS), 


where  OS  and  NS  represent  old  state  and 
new  suite.  Although  the  external  behavior 
of  a  state  box  is  identical  to  its  correspond¬ 
ing  black  box,  die  stimulus  histories  are 
replaced  with  references  to  an  old  state 
and  the  generation  of  a  new  suite,  as  its 
transitions  require. 

As  in  the  traditional  view  of  objects, 
state  boxes  encapsulate  state  data  and  ser¬ 
vices  (methods)  on  that  daui.  In  this  view, 
stimuli  and  responses  are  inputs  and  out¬ 
puts,  respectively,  of  specific  state-box  ser¬ 
vice  invocations  that  operate  on  suite  data. 

dear  box.  An  object’s  clear  box  is  derived 
from  its  suite  box  by  defining  a  procedure 
to  carry  out  the  state-box  transition  func¬ 
tion.  The  transition  function  of  a  clear  box 
is 

(S,  OS)  (R,  NS)  by  procedure 

So  a  clear  box  is  simply  a  program  that 
implements  the  corresponding  state  box. 
A  clear  box  may  invoke  black  boxes  at  the 
next  level,  so  the  refinement  process  is  re¬ 
cursive,  with  each  clear  box  possibly  intro¬ 
ducing  opportunities  for  defining  new  ob¬ 
jects  or  extensions  to  existing  ones. 

Clear  boxes  play  a  crucial  role  in  the 
usage  hierarchy  by  ensuring  the  harmoni¬ 
ous  cooperation  of  objects  at  the  next  level 
of  refinement  Objects  and  their  dear-box 
I  connections  are  derived  from  immediate 


processing  needs  at  each  stage  of  refine¬ 
ment,  not  invented  a  priori,  with  uncer¬ 
tain  connections  left  to  be  defined  later. 
The  design  and  verification  ofelear-box 
procedures  is  the  focus  of  die  next  sec¬ 
tion. 

Because  state  boxes  can  be  verified 
with  respect  to  their  black  boxes  and  clear 
boxes  with  respect  to  their  state  boxes,  box 
structures  bring  correctness  verification  to 
object  architectures.** 

DESIGN  AND  VERIFICATION 

The  procedural  control  structures  of 
structured  programming  used  in  clear- 
box  design  —  sequence,  alternation  (if- 
then-else),  and  iteration  (while-do)  —  are 
single-entry,  single-exit  structures  that 
cannot  produce  side  effects  in  control 
flow.  (Control  structures  for  concurrent 
execution  are  dealt  with  in  box  structures, 
but  are  outside  the  scope  of  this  article.) 

When  it  executes,  a  given  control 
structure  simply  transforms  data  from  an 
input  state  to  an  output  state.  This  trans¬ 
formation,  known  as  its  pivgram  function , 
corresponds  to  a  mathematical  function:  It 
defines  a  mapping  from  a  domain  to  a 
range  by  a  particular  rule. 

For  integers  w,  x,  y,  and  z,  for  example, 
die  program  function  of  die  sequence, 
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Sequence 

Control  Correctness 

structure:  condition  (for  al  arguments): 


-DO 

. -  -  . 

g; 

Does  g  followed 

h 

by/idof? 

END 

Alternation 

Control 

Correctness 

structure: 

condition: 

tf] 

IF  p 

Whenever  pis  true 

THEN 

does  g  do  ( ond 

g 

whenever  pis  false 

ELSE 

does  h  do  f? 

h 

END 

Iteration 

Control 

Correctness 

structure: 

condition: 

[f  3 

Is  termination 

while  p  guoronteed,  ond 

DO 

whenever  pis  true 

g 

does  g  followed 

END 

by  fdo  ( ond 
whenever  p  is  folse 

does  doing 
nothing  do  f? 

DO 

z  :  =  abs  {y ) 
w  : =  max (x,  z ) 
END 


is,  in  concurrent  assignment  form, 

w,  z  :=  max  (x,  abs(y)),  abs(y) 

For  integer  x  >  0 ,  die  program  func¬ 
tion  of  die  iteration 


END 

is,  in  English, 

set  odd  X  to 


Figure  3.  Correctness  conditions  in  ques¬ 
tion  form  for  verifying  each  type  of  dear- 
box  control  structure. 


Design  refinement.  In  designing  dear-box 
procedures,  you  define  an  intended  func¬ 
tion,  then  refine  it  into  a  control  structure 
and  new  intended  functions,  as  Figure  2 
illustrates.  Intended  functions,  enclosed  in 
braces,  are  recorded  in  die  design  and  at¬ 
tached  to  their  control -structure  refine¬ 
ments.  In  essence,  clear  Ixxxes  am  comjxised 
of  a  finite  number  of  control  structures,  each 
of  which  can  lie  checked  for  correctness. 

Design  simplification  is  an  important 
objective  in  die  stepwise  refinement  of 
clear  boxes.  The  goal  is  to  generate  com¬ 
pact,  straightforward,  verifiable  designs. 


Program: 

[f  13 
DO 

gi 

g2 
[  f  2  ] 

WHILE 

Pi 

DO  [  f  3  ] 
g3 
[f4] 

IF 

p2 

THEN  [  f 5] 

g4 

g5 

ELSE  [  f 6 ] 
g6 

gv 

END 

g8 

END 

END 


Subproofs : 

fl  =:  [DO  gl  ;g2  ;  [  f 2 3  END]  ? 

f  2  =  [WHILE  pi  DO  [  f  3)  END]  ? 

f 3  =  [DO  g3 ;  [f 4 ] ;g8  END]  ? 
f 4  =  [IF  p2  THEN  [f  5]  ELSE  [  f  6 ]  END] 

f 5  =  [DO  g4;g5  END]  ? 
f 6  =  [DO  g6;g7  END]  ? 


Figure  4 .  A  clear-box  procedure  and  its  constituent  subproofs.  In  the  figure ,  each  pi  is  j 
a  predicate ,  each  gi  is  an  operation ,  and  each  f  i  is  an  intended  function.  ’ 


Correctness  verification.  To  verify  die  cor¬ 
rectness  of  each  control  structure,  you  de¬ 
rive  its  program  function  —  die  function  it 
actually  computes  —  and  compare  it  to  its 
intended  function,  as  recorded  in  die  de¬ 
sign.  A  correctness  theorem'  defines  how 
to  do  diis  comparison  in  terms  of  lan¬ 
guage-  and  application-independent  cor¬ 
rectness  conditions ,  which  you  apply  to  each 
control  structure. 

Figure  3  shows  die  correctness  condi¬ 
tions  for  the  sequence,  alternation,  and  it¬ 
eration  control  structures.  Verifying  a  se¬ 
quence  involves  function  composition  and 
requires  checking  exactly  one  condition. 
Verifying  an  alternation  involves  case 
analysis  and  requires  checking  exactly  two  ; 
conditions.  Verifying  an  iteration  involves  • 
function  composition  and  case  analysis  in 
a  recursive  equation  and  requires  checking  ' 
exaedy  diree  conditions. 

Correctness  verification  has  several  ad-  j 
Mintages:  j 

♦  It  reduces  verification  to  a  finite  process.  \ 
As  Figure  4  illustrates,  the  nested,  se-  j 
quenced  way  dvat  control  structures  are  j 
organized  in  a  clear  box  naturally  defines  a  j 
hierarchy  diat  reveals  die  correctness  con-  j 
ditions  diat  must  be  verified.  An  axiom  of  j 
replacement7  lets  us  substitute  intended 
functions  for  dieir  cono'ol  structure  re¬ 
finements  in  die  hierarchy  of  subproofs. 
For  example,  the  subproof  for  the  in¬ 
tended  function  fl  in  Figure  4  requires 
proving  diat  die  composition  of  opera¬ 
tions  gl  and  g2  widi  intended  subfunc-  i 
tion  f  2  has  the  same  effect  on  data  as  f  1. 
Note  that  f  2  substitutes  for  all  die  details 
of  its  refinement  in  this  proof.  This  substi¬ 
tution  localizes  die  proof  argument  to  the 
control  structure  at  hand.  In  fact,  it  lets 
you  carry  out  proofs  in  any  order. 

It  is  impossible  to  overstate  the  positive 
effect  that  reducing  verification  to  a  fi¬ 
nite  process  has  on  quality.  Even  though 
all  but  the  most  trivial  programs  exhibit 
an  essentially  infinite  number  of  execu¬ 
tion  paths,  they  can  be  verified  in  a  finite 
number  of  steps.  For  example,  the  clear 
box  in  Figure  5  has  exactly  15  correct¬ 
ness  conditions  that  must  be  verified. 

♦  It  lets  Cleanroom  teams  verify  every 
line  of  design  and  code.  Teams  can  cany 
out  the  verification  through  group  j 
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JTo  :=  odd_numbers  (Q)  II  even_numbers  (Q)  3 

PROC  Odd_Before_Even  (ALT  Q} 


seq 

1 


DATA 

odds 

:  queue 

of  integer  [initializes 

to  empty] 

evens 

:  queue 

of  integer  [initializes 

to  empty] 

X 

:  integer 

END 

[  C 

=  empty. 

— 

odds 

=  odds 

|odd_numbers (Q) , 

evens 

=  evens 

I even_numbe  r  s ( Q )  3 

WHILE  Q  <>  empty 
DO 

x  :=  end(Q) 

;x  is  odd  ->  odds  :  =  odds  II  x 
true  ->  evens  :=  evens  II  x 
I:  odd(x) 

THEN 

end (odds)  :=  x 
end (evens)  :=  x 

END 


END 

[  c  : =  Q  I  I  odds , 
odds  : =  empty  ] 

WHILE  odds  <>  empty  _ , 

DO  ;er.d(Q)  :=  end  (odds) 

>:  :=  end  (odds) 

end (Q)  :=  x 

END 


seq— i 


wdo 

3 


seq 

1 


wdo 

3 


[  C  : =  Q  I !  evens , 

evens :=  empty] 

WHILE  evens  <>  empty  _ i 

DO  ;end(Q)  :  =  end (evens) 

X  :=  end (evens) 

end (Q)  :=  x 


seq 

1 


wdo 

3 


END 

END  OCd- 


:etore-even 


Figure  5.  A  clear-box  procedure  with  15  correctness  conditions  to  be  verified.  The 
procedural  control  structures  and  the  number  of  correctness  conditions  that  must  be 
checked  are  shown  in  bold.  Seq  indicates  a  sequence,  ite  indicates  an  alternation  (ifthen- 
else),  and  wdo  indicates  an  iteration  (while-do). 


analysis  and  discussion  on  the  basis  ofthe 
correctness  theorem,  and  they  can  pro¬ 
duce  written  proofs  when  extra  confidence 
in  a  life-  or  mission-critical  system  is  re¬ 
quired. 

♦  It  results  in  a  near-zero  defect  level. 
During  a  team  review,  every  correctness 
condition  of  every  control  structure  is  ver¬ 
ified  in  turn.  Every  team  member  must 
agree  that  each  condition  is  correct,  so  an 
error  is  possible  only  if  ever)7  team  mem¬ 
ber  incorrectly  verifies  a  condition.  The 
requirement  for  unanimous  agreement 
based  on  individual  verifications  results 
in  software  that  has  few  or  no  defects  be¬ 
fore  fust  execution. 

♦  It  scales  up.  Every  software  system,  no 
matter  how  large,  has  top-level,  clear-box 
procedures  composed  of  sequence,  alter¬ 
nation,  and  iteration  structures.  Each  of 
these  typically  invokes  a  large  subsystem 
with  thousands  of  lines  of  code  —  and  each 
of  those  subsystems  has  its  own  top-level 
intended  functions  and  procedures.  So  die 
correctness  conditions  for  these  high-level 
control  structures  are  verified  in  die  same 
way  as  are  diose  of  low-level  structures. 
Verifi cation  at  high  levels  may  take,  and 
well  be  worth,  more  time,  but  it  does  not 
take  more  dieory. 

♦  It  produces  better  code  than  unit  testing 
Unit  testing  checks  only*  the  effects  of 
executing  selected  test  paths  out  of  many 
possible  paths.  By  basing  verification 
on  function  theory,  the  Cleanroom  ap¬ 
proach  can  verify7  every7  possible  effect 
on  all  data,  because  while  a  program 
may  have  many  execution  paths,  it  has 
only  one  function.  Verification  is  also 
more  efficient  than  unit  testing.  Most 
verification  conditions  can  be  checked  in  a 
few  minutes,  but  unit  tests  tike  substantial 
time  to  prepare,  execute,  and  check. 

QUALITY  CERTIFICATION 

Statistical  quality7  control  is  used  when 
you  have  too  many  items  to  test  all  of  them 
exhaustively7.  Instead,  you  statistically 
sample  and  analyze  some  items  to  obtain  a 
scientific  assessment  of  the  quality  of  all 
items.  This  technique  is  widely  used  in 
manufacturing,  in  which  items  on  a  pro¬ 
duction  line  are  sampled,  their  quality7  is 


measured  against  a  presumably  perfect  de¬ 
sign,  the  sample  quality7  is  extrapolated  to 
die  entire  production  line,  and  flaws  in 
production  are  corrected  if  the  quality  is 
too  low. 

In  hardware  products,  the  statistics 
used  to  establish  quality  are  derived  from 
slight  variations  in  the  products5  physical 
properties.  But  software  copies  are  identi¬ 
cal,  bit  for  bit.  What  statistics  can  we  sam¬ 


ple  to  extrapolate  quality? 

Usage  testing.  It  turns  out  that  software 
has  a  statistical  property  of  great  interest  to 
developers  and  users  —  its  execution  be¬ 
havior.  How  long,  on  average,  will  a  soft¬ 
ware  product  execute  before  it  fails? 

From  this  notion  has  evolved  the  pro¬ 
cess  of statistical  usage  testing ,8  in  which  you 
♦  sample  the  (essentially  infinite)  pop- 
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CLEANROOM  QUALITY  RESULTS 

Cleanroom  projects  report 
a  testing  arm-  rate  pa'  thousand 
lines  of  code,  which  represents 
residual  errors  in  the  software 
after  correctness  verification. 
The  projects  briefly  described 
here  are  among  1 7  Cleanroom 
projects,  involving  nearly  a  mil¬ 
lion  lines  of  code,  that  have  re¬ 
ported  a  weighted  average  of  23 
arois per  KLOC  found  in  all  test¬ 
ing ,  measured foom  Jhst-ever  exe- 
aition  of  the  code  —  a  remark¬ 
able  quality  achievement1 

♦  IBM  Cobol  Structuring  Fa¬ 
cility  (Cobol/SF).  This  was 
IBM’s  first  commercial  Clean- 
room  product,  developed  by  a 
six-person  team.  This  85 
KLOC  PL/I  program  automat¬ 
ically  transforms  un¬ 
structured  Cobol  programs 
into  functionally  equivalent 
structured  form  for  im¬ 
proved  understandability 
and  maintenance.  It  had  a 
testing  error  rate  of  3.4  er¬ 
rors  per  KLOC;  several 
major  components  completed 
certification  with  no  errors 
found.  In  months  of  intensive 
beta  testing  at  a  major  aero¬ 
space  corporation,  all  Cobol 
programs  executed  identically 
before  and  after  structuring. 

Productivity,  including  all 
specification,  design,  verifica¬ 
tion,  certification,  user  publica¬ 
tions,  and  management,  aver¬ 
aged  740  LOC  per  person- 
month.  So  far,  a  small  fraction 
of  a  person-year  per  year 
has  been  required  for  all 
maintenance  and  customer 
support.  Although  the  prod¬ 
uct  exhibits  a  complexity 
level  on  the  order  of  a  Cobol 
compiler,  just  seven  minor 
errors  were  reported  in  the 
first  three  years  of  field  use,  all 
resulting  in  simple  fixes.  — R.C. 
Linger  and  HD.  Mills,  “A  Case 
Study  in  Cleanroom  Software 
Engineerings  The  IBM  Cobol 
Structuring  Facility,”  Proc.  Com - 
psac ,  IEEE  CS  Press,  Los  Al- 
amitos,  Calif.,  1988,  pp.  10-17. 


♦  NASA  satellite-control  proj¬ 
ect .  The  Coarse/Fine  Attitude 
Determination  System 
(CFADS)  of  the  NASA  Atti¬ 
tude  Ground  Support  System 
(AGSS)  was  the  first  Clean¬ 
room  project  carried  out  by  the 
Software  Engineering  Labora¬ 
tory  of  the  NASA  Goddard 
Space  Flight  Center.  The  sys¬ 
tem,  comprising  40  KLOC  of 
Fortran,  exhibited  a  testing 
error  rate  of  4.5  errors  per 
KLOC.  Productivity  was  780 
LOC  per  person-month,  an  80 
percent  improvement  over  pre¬ 
vious  SEL  averages.  Some  60 
percent  of  the  programs  com¬ 
piled  correctly  on  the  first  at¬ 
tempt.  —  A  Kouchakdjian,  S. 
Green,  and  V.R.  Basili,  “Evalua¬ 
tion  of  the  Cleanroom  Method¬ 
ology  in  the  Software  Engi¬ 
neering  Laboratory,”  Proc.  14th 
Sojtwai-e  Eng.  Workshop ,  NASA 
Goddard  Space  Flight  Center, 
Greenbelt,  Md.,  1989. 

♦  Martin  Marietta  Auto¬ 
mated  Documentation  System. 

A  four-person  Cleanroom 
team  developed  die  prototype 
for  this  system,  a  1,820-line  re¬ 
lational  database  application 
written  in  Foxbase.  It  had  a 
testing  error  rate  of  0.0  er¬ 
rors  per  KLOC  —  no  com¬ 
pilation  errors  were  found 
and  no  failures  w'ere  encoun¬ 
tered  in  statistical  testing 
and  quality  certification. 

The  software  was  certified 

at  target  levels  of  reliability 
and  confidence.  Team  mem¬ 
bers  attributed  error-free 
compilation  and  failure-free 
testing  to  the  rigor  of  the 
Cleanroom  method.  —  C.J. 
Trammell,  L.H.  Binder,  and 
C.E.  Snyder,  “The  Automated 
Production  Control  System:  A 
Case  Study  in  Cleanroom  Soft¬ 
ware  Engineering,MCM 
Trans.  Software  Eng.  and  Meth¬ 
odology, Jan.  1992,  pp.  81-94. 

♦  IBM  AOEXPERT/MVS. 

A  50-person  team  developed 


this  complex  decision-support 
facility  that  uses  artificial  intelli¬ 
gence  to  predict  and  prevent 
operating  problems  in  an 
MVS  environment.  The  sys¬ 
tem,  written  in  PL/I,  C,  Rexx, 
and  TIRS,  totaled  107 
KLOC,  developed  in  three  in¬ 
crements.  It  had  a  testing 
error  rate  of  2.6  errors  per 
KLOC.  Causal  analysis  of  the 
first  16-KLOC  increment  re¬ 
vealed  that  five  of  its  eight 
components  experienced  no  er¬ 
rors  in  testing. 

The  project  reported  devel¬ 
opment  team  productivity  of 
486  LOC  per  person-month. 
No  operational  errors  have 
been  reported  to  date  from 
beta  test  and  early  user  sites.  — 
PA  Hausler,  “A  Recent  Clean¬ 
room  Success  Story:  The  Red¬ 
wing  Project,”  Proc.  1 7th  Soft - 
iv me  Eng.  Workshop ,  NASA 
Goddard  Space  Flight  Center, 
Greenbelt,  Md.,  1992. 

♦  NASA  satellite-control  pro¬ 
jects.  Two  satellite  projects,  a  20- 
KLOC  attitude-determination 
subsystem  and  a  1 50-KLOC 
flight-dynamics  system,  were 
the  second  and  third  Clean¬ 
room  projects  undertaken  at 
NASA’s  Software  Engineering 
Laboratory.  These  systems  had 
a  combined  testing  error  rate  of 
4.2  errors  per  KLOC.  —  S.E. 
Green  and  Rose  Pajersld, 
“Cleanroom  Process  Evolution 
in  the  SEL,”  Proc.  16th  Software 
Eng.  Wot'kshop ,  NASA  God¬ 
dard  Space  Flight  Center, 
Greenbelt,  Md.,  1991. 

♦  IBM  3090E  tape  drive.  A 
five-person  team  developed  the 
device -controller  design  and 
microcode  in  86  KLOC  of  C, 
including  64  KLOC  of  func¬ 
tion  definitions.  This  em¬ 
bedded  software  processes 
multiple  real-time  I/O  data 
streams  to  support  tape- 
cartridge  operations  in  a 
multibus  architecture.  The 
box-structure  specification 


for  the  chip-set  semantics 
revealed  several  hardware 
errors.  The  project  had  a 
testing  error  rate  of  1.2  er¬ 
rors  per  KLOC. 

-A-one-module  experiment 
compared  the  effectiveness  of 
unit  testing  and  correctness 
verification.  In  unit  testing, 
the  team  took  10  person-days 
to  develop  scaffolding  code, 
invent  and  execute  test  cases, 
and  check  results.  They  found 
seven  errors.  Correctness  veri¬ 
fication,  which  required  an 
hour-and-a-half  in  a  team  re¬ 
view',  found  the  same  seven  er¬ 
rors  plus  three  more. 

To  meet  a  business  need, 
the  third  code  increment  went 
straight  from  development, 
with  no  testing  whatsoever, 
into  customer-evaluation  dem¬ 
onstrations  usinglive  data. 
There  were  no  errors  of  any 
kind.  A  total  of 490  statistical 
tests  were  executed  against  the 
final  version  of  the  system,  with 
no  errors  found. 

♦  Ericsson  Telecom  OS32  op¬ 
erating  systan.  This  70-person, 

1 8-month  project  specified,  de¬ 
veloped,  and  certified  a  3 50- 
KLOC  operating  system  for  a 
new  family  of  switching  comput¬ 
ers.  The  project  had  a  testing 
error  rate  of  1.0  errors  per 
KLOC. 

Productivity  was  reported 
to  have  increased  by  70  per¬ 
cent  for  development;  100 
percent  for  testing.  The  team 
significantly  reduced  develop¬ 
ment  time,  and  the  project 
was  honored  by  Ericsson  for 
its  contribution  to  the  com¬ 
pany.  —  L.-G.  Tann,  “OS32 
and  Cleanroom,”  Pi'oc.  IstEinv- 
pean  Industrial  Symp.  Clean¬ 
room  Software  Eng.,  Q-labs, 
Lund,  Sweden,  1993. 
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Figure  6.  (A)  Simplified  usage  probability  distribution  for  a  program  with  four  use? 
stimuli  and  (B)  a  sample  of  associated  test  cases. 


Figure  7.  Two  sample  graphs.  The  curve  for  high-quality  software  shows  exponential 
improvement ,  such  that  the  MTTF  quickly  exceeds  the  total  test  time.  The  anise  for 
low -quality  software  shows  little  MTTF  growth. 


ulation  of  all  possible  executions  (correct 
and  incorrect)  by  users  (people  or  other 
programs)  according  to  how  frequently 
you  expect  the  executions  to  happen, 

♦  measure  their  quality  by  determin¬ 
ing  if  the  executions  are  correct, 

♦  extrapolate  die  quality  of  die  sample 
to  die  population  of  possible  executions, 
and 

♦  identify  and  correct  flaws  in  the  de¬ 
velopment  process  if  the  quality  is  inade¬ 
quate. 

Statistical  usage  testing  amounts  to 
testing  software  die  way  users  intend  to 
use  it.  The  entire  focus  is  on  external  sys¬ 
tem  behavior,  not  die  internals  of  design 
and  implementation.  CJeanroom  certifi¬ 
cation  teams  have  deep  knowledge  of  ex¬ 
pected  usage,  but  require  no  knowledge  of 
design  internals.  Their  role  is  not  to 
debug-in  quality,  an  impossible  task,  but 
to  scientifically  certify  software’s  quality 
through  statistical  testing. 

In  practice,  Cleanroom  quality  certifi¬ 
cation  proceeds  in  parallel  with  develop¬ 
ment,  in  three  steps. 

1.  Specif'  usage-probability  distributions. 
Usage-probability  distributions  define  all 
possible  usage  patterns  and  scenarios,  in¬ 
cluding  erroneous  and  unexpected  usage, 
togedier  widi  dieir  probabilities  of  occur¬ 
rence.  The)7  are  defined  on  die  basis  of  die 
functional  specification  and  odier  sources 
of  information,  including  interviews  widi 
prospective  users  and  die  pattern  of  use  in 
prior  versions. 

Figure  6a  shows  a  usage  specification 
fora  program  with  four  user  stimuli:  up¬ 
date  (U),  delete  (D),  query  (Q),  and 
print  (P).  A  simplified  distribution  that 
omits  scenarios  of  use  and  other  details 
shows  projected  use  probabilities  of 
32,  14,  46,  and  8  percent,  respective!)7. 
These  probabilities  are  mapped  onto  an 
interval  of  0  to  99,  dividing  it  into  four 
partitions  proportional  to  the  probabili¬ 
ties.  Usage-probability  distributions  for 
large-scale  systems  are  often  recorded  in 
formal  grammars  or  Markov  chains  for 
analysis  and  automatic  processing. 

In  incremental  development,  you  can 
stratify  a  usage-probability  distribution 
into  subsets  that  exercise  increasing 
functional  content  as  increments  are 


added,  widi  the  full  distribution  in  effect 
once  the  final  increment  is  in  place.  In  ad¬ 
dition,  you  can  define  alternate  distribu¬ 
tions  to  certify7  infrequently  used  system 
functions  whose  failure  has  important 
consequences,  such  as  the  code  for  a  nu¬ 
clear-reactor  shutdown  system. 

2.  Derive  test  cases  that  are  ran  dam  ly  gen¬ 
erated  from  usage-probability  distributions. 
Test  cases  are  derived  from  the  distribu¬ 
tions,  such  that  every  test  represents  ac¬ 
tual  use  and  will  effectively  rehearse  user 
experience  with  die  product.  Because 
test  cases  are  completely  prescribed  by 
the  distributions,  producing  them  is  a  me¬ 
chanical,  automatable  process. 

Figure  6b  show  s  test  cases  for  the  prob¬ 
ability  distribution  in  Figure  6a.  If  you  as¬ 
sume  a  test  case  contains  six  stimuli,  then 


you  generate  each  test  by  obtaining  six 
two-digit  random  numbers.  These  num¬ 
bers  represent  the  partition  in  which  the 
corresponding  stimuli  (U,  D,  Q,  or  P)  re¬ 
sides.  In  this  way,  each  test  case  is  faidiful 
to  die  distribution  and  represents  a  possi¬ 
ble  user  execution.  For  testing  large-scale 
systems,  usage  grammars  or  Markov 
chains  can  be  processed  to  generate  test 
cases  automatically. 

3.  Execute  test  cases ,  assess  results ,  and 
compute  quality  measures.  At  this  point,  die 
development  team  has  released  verified 
code  to  the  certification  team  for  first-ever 
execution.  The  certification  team  executes 
each  test  case  and  checks  the  results 
against  system  specifications.  The  team 
records  execution  time  up  to  die  point  ol 
any  failure  in  appropriate  units,  for  exam- 


pie,  CPU  time,  wall-clock  time,  or  num¬ 
ber  of  transactions. 

These  into  fail  times  represent  the  qual¬ 
ity  of  the  sample  of  possible  user  execu¬ 
tions.  They  are  passed  to  a  quality  certifi¬ 
cation  model  that  computes  the  system’s 
quality,  including  its 
mean  time  to  failure.  The 
quality-certification 
model  produces  graphs 
like  the  one  in  Figure  7. 

Because  the  Clean- 
room  development  pro¬ 
cess  rests  on  a  formal,  sta¬ 
tistical  design,  these 
M  ITT  measures  provide 
a  scientific  basis  for  man¬ 
agement  action,  unlike  the  anecdotal  evi¬ 
dence  from  coverage  testing  (If  few  errors 
are  found,  is  that  good  or  bad?  If  many 
errors  are  found,  is  that  good  or  bad?).  In 
theory,  there  is  no  way  to  ever  know  that  a 
software  system  has  zero  defects.  How¬ 
ever,  as  failure- free  executions  accumu¬ 
late,  it  becomes  possible  to  conclude  that 
the  software  is  at  or  near  zero  defects  with 
high  probability. 

Extending  MTTF.  But  there  is  more  to  the 
story  of  statistical  usage  testing.  Extensive 


analysis  of  errors  in  large-scale  software 
systems  reveals  a  spread  in  the  failure  rates 
of  errors  of  some  four  orders  of  magni¬ 
tude.9  Virulent,  high-rate  errors  can  liter¬ 
ally  occur  every  few  hours  for  some  users, 
but  low-rate  errors  may  show  up  only  after 
accumulated  decades  of  use 
by  many  users. 

High-rate  errors  are 
responsible  for  nearly 
two-thirds  of  software 
failures  reported,10  even 
though  they  comprise  less 
than  three  percent  of  total 
errors.  Because  statistical 
usage  testing  amounts  to 
testing  software  die  way 
users  will  use  it,  high-rate  errors  tend  to  be 
found  first.  Any  errors  left  behind  after 
testing  tend  to  be  infrequently  encoun¬ 
tered  by  users. 

Traditional  coverage  testing  finds  er¬ 
rors  in  random  order.  Yet  finding  and 
fixing  low-rate  errors  has  little  effect  on 
MTTF  and  user  perception  of  quality, 
while  finding  and  fixing  errors  in  failure- 
rate  order  has  a  dramatic  effect.  Statisti¬ 
cal  usage  testing  is  far  more  effective 
than  coverage  testing  at  extending 
MTTF.10 


Software  that  is  formally  engineered  in 
increments  is  well-documented  and 
under  intellectual  control  throughout  de¬ 
velopment.  The  Cleanroom  approach 
provides  a  framework  for  managers  to 
plan  (and  replan)  schedules,  allocate  re¬ 
sources,  and  systematically  accommodate 
changes  in  functional  content. 

Experienced  Cleanroom  teams  can 
substantially  reduce  time  to  market.  This 
is  due  largely  to  the  precision  imposed  on 
development,  which  helps  eliminate  re¬ 
work  and  dramatically  reduces  testing 
time,  compared  with  traditional  methods. 
Furthermore,  Cleanroom  teams  are  not 
held  hostage  by  error  correction  after  re¬ 
lease,  so  they  can  initiate  new  develop¬ 
ment  immediately. 

The  cost  of  quality  is  remarkably  low  in 
Cleanroom  operations,  because  it  mini¬ 
mizes  expensive  debugging  rework  and 
retesting. 

Cleanroom  technology  builds  on  exist¬ 
ing  skills  and  software-engineering  prac¬ 
tices.  It  is  readily  applied  to  both  new  sys¬ 
tem  development  and  reengineering  and 
extending  legacy  systems.  As  die  need  for 
higher  quality  and  productivity  in  soft¬ 
ware  development  increases,  we  believe 
diat  use  of  die  Cleanroom  process  will 
continue  to  grow.  ♦ 
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Abstract 


This  paper  presents  the  results  and  lessons  learned  from  a  STARS  (Software  Technology  for  Adaptable, 
Reliable  Systems)  sponsored  process  technology  transfer  demonstration.  In  March  of  1992,  the  Armament, 
Munitions  and  Chemical  Command  (AMCCOM),  now  the  Tank-automotive  and  Armaments  Command 
(TACOM),  Life  Cycle  Software  Engineering  Center  (LCSEC)  at  Picatinny  Arsenal  was  selected  to  demonstrate 
that  Cleanroom  Software  Engineering  (CSE)  can  be  successfully  applied  in  a  typical  DoD  Software  Support 
Activity  (SSA).  Results  indicate  that: 

•  CSE  practices  can  be  successfully  transferred  to  a  typical  DoD  SSA, 

•  engineering  staff  productivity  and  product  quality  were  increased  while  simultaneously  increasing  job 
satisfaction,  and 

•  a  return  on  investment  of  at  least  1 1 : 1  has  been  realized  on  the  first  project  to  which  CSE  techniques  were 

applied. 


Introduction 

The  goal  for  the  technology  transfer  effort  for  the  LCSEC  at  Picatinny  Arsenal  was  to  conduct  a 
demonstration  of  CSE  practices  at  a  typical  DoD  SSA. 

The  LCSEC  was  selected  in  response  to  their  expressed  interest  in  improving  the  process  by  which  they 
maintain  software  in  general  and,  specifically,  in  using  the  CSE  technology.  Additionally,  as  a  typical  DoD  SSA, 
it  was  deemed  important  to  improve  the  processes  on  which  the  US  Army  spends  the  largest  portion  of  their 
software  money;  i.e.,  in  software  maintenance  and  re-engineering  (as  opposed  to  new  software  development).  The 
demonstration  was  facilitated  by  Loral  and  SET  (Software  Engineering  Technology,  Inc.). 

The  LCSEC  at  Picatinny  Arsenal  is  a  representative  DoD  Software  Support  Activity  that  wants  to  apply  a 
more  effective  approach  to  software  support.  The  current  state  of  software  re-engineering  at  the  LCSEC  varies 
from  project  to  project  but  the  majority  have  not  achieved  the  desired  level  of  productivity  and  quality.  A  major 
goal  of  the  Picatinny  Arsenal  LCSEC  is  to  achieve  a  Software  Engineering  Institute  Capability  Maturity  Model 
(SEI CMM)  Level  3  rating  by  adopting  an  evolutionary  process  improvement  approach  to  software  re¬ 
engineering.  Currently,  the  LCSEC  is  receiving  support,  under  STARS  Task  IA02  from  Loral  and  SET,  in 
applying  the  Cleanroom  approach  on  the  re-engineering  of  the  Mortar  Ballistic  Computer  (MBC)  into  the 
Improved  MBC  (I-MBC).  Initial  results  have  been  successful,  the  project  personnel  are  employing  the 
Cleanroom  engineering  practices  and  adopting  the  process  driven,  team  oriented  approach. 

DoD  Software  Support  Activities  (SSAs)  provide  important  opportunities  to  demonstrate  STARS  efforts 
to  improve  software  quality  and  productivity.  SSA  activities  represent  a  major  portion  of  the  DoD  software 


budget  and  the  proportion  is  expected  to  be  increased  during  the  next  decade.  This  will  occur  as  the  many 
systems  in  the  DoD  development  pipeline  are  turned  over  to  SSAs  for  support.  It  is  likely  that,  as  fewer  new 
systems  come  into  the  inventory,  DoD  managers  will  attempt  to  extend  the  useful  life  of  old  systems  through 
software  enhancements  and  re-engineering. 


STARS  Program 

The  STARS  program  is  a  DoD  research  and  development  effort  funded  under  the  Advanced  Research 
Projects  Agency  (ARPA).  The  main  thrust  of  this  effort  is  that  software  engineering  is  process  driven,  domain- 
specific,  reuse-based,  and  supported  by  an  integrated  software  engineering  environment.  This  concept  is  called 
Megaprogramming  and  the  STARS  program  is  currently  engaged  in  several  demonstration  projects  of  the 
technologies  developed  earlier  in  the  program.  The  Picatinny  MBC  effort  was  the  first  demonstration  project  to 
use  STARS  concepts.  Loral  is  one  of  the  prime  contractors  for  this  effort  and  SET  is  a  principal  subcontractor  for 
the  Loral/STARS  effort. 


TACOM  LCSEC  Overview 

The  LCSEC  at  Picatinny  Arsenal  provides  a  number  of  services  including:  software  acquisition  support  to 
program  managers,  computer  resource  life  cycle  management  plans,  pre-planning  for  software  support,  manage 
contracted  post  deployment  software  support  efforts,  software  configuration  management,  and  design  and 
implement  software  changes.  The  types  of  battlefield  automated  systems  supported  include  cannon  and  tank  gun 
systems;  smart  mines  and  munitions;  ballistics  computers;  gunnery  simulators;  trainers;  and  nuclear,  biological, 
and  chemical  detection  systems. 

The  desire  for  process  driven  technology  was  the  result  of  a  Software  Process  Assessment  (SPA) 
conducted  by  a  team  of  representatives  from  the  AMC  LCSECs  with  coaching  from  the  Software  Engineering 
Institute  (SEI).  Picatinny  LCSEC  management  has  developed  a  close  relationship  with  the  SEI  because  they 
desired  help  with  identifying  areas  to  achieve  the  desired  level  of  productivity  and  quality.  Review  of  the  SPA 
findings  lead  LCSEC  management  to  realize  that  the  software  engineering  process  was  not  under  intellectual 
control.  Each  new  software  project,  whether  performed  by  contractors  or  civil  servants,  was  treated  largely  as 
new  activity  that  did  not  necessarily  draw  on  prior  experience  for  process  improvement.  The  only  factor  that 
perpetuated  experience  was  people,  be  it  government  or  contractor,  who  participated  in  the  same  projects  time 
after  time.  Documentation  received  by  Picatinny,  when  they  were  given  systems  to  maintain,  was  poor  or  not  up 
to  date  and  no  defined  process  existed  for  maintaining  continual  project  control.  In  other  words,  the  state-of-the- 
practice  consisted  of  traditional  software  engineering  practices  that  were  ad-hoc  in  nature,  as  opposed  to  a 
disciplined,  defined  software  engineering  process.  These  realizations  and  the  results  from  the  SPA  were  the  basis 
for  their  move  to  enhance  their  software  engineering  capabilities. 

Typical  DoD  SSA  organizations  have  immature  processes  and  are  subject  to  morale  problems  among 
software  engineers  due  to  the  combination  of  an  undefined  manner  of  doing  work,  along  with  a  lack  of  task- 
oriented  scheduling.  The  software  engineers  at  the  LCSEC  did  their  work  well  because  of  individual  skills,  but 
often  seemed  to  be  stuck  in  the  same  "groove,"  where  the  same  situations,  in  terms  of  schedule,  would  arise  year 
after  year.  A  general  lack  of  enthusiasm  pervaded  initial  discussions  with  project  teams. 

Despite  these  difficulties,  however,  the  customers  (various  users  within  the  US  Army)  indicate  that  they 
are  basically  content  with  the  quality  of  the  products.  Not  many  field  reports  of  failures  are  submitted  by  their 
customers,  due  to  extensive,  pre-release  user  testing.  Unfortunately,  evidence  suggests  that  this  may  also  result 
from  the  absence  of  formal  failure  observation  and  reporting  mechanisms,  making  the  field  quality  of  developed 
products  difficult  to  ascertain. 


LCSEC  management  at  Picatinny  recognized  the  problems  with  their  state  of  the  practice  and  took  the 
initiative  to  recognize  Cleanroom  Software  Engineering  (CSE)  as  the  mechanism  with  which  to  facilitate  the 
desired  cultural,  technical  and  process  changes. 


Cleanroom  Software  Engineering  (CSE) 

CSE  was  chosen  as  the  process  driven  technology  because  it  addresses  the  deficiencies  identified  during 
the  LCSEC  Software  Process  Assessment  (SPA).  CSE's  management  and  development  team  approach  was 
consistent  with  quality  management  philosophy,  e.g.  workforce  empowerment,  process  focus,  and  quantitative 
orientation.  It  provides  for  the  transition  of  process  technology  to  the  project  staff  and  integrates  several  proven 
software  engineering  practices  into  one  methodology.  LCSEC  management  anticipated  productivity  gains  and 
morale  enhancement  from  the  introduction  of  the  technology. 

CSE  consists  of  a  body  of  practical  and  theoretically  sound  engineering  principles  applied  to  the  activity 
of  software  engineering.  Cleanroom  consists  of  a  thorough  specification  phase;  resulting  in  a  six  part 
specification,  including  a  precise,  black  box  description  of  the  software  part  of  a  system.  Software  development 
proceeds  from  the  black  box  specification  via  a  step-wise  refinement  procedure  using  box-structured  design 
concepts.  This  process  focuses  on  defect  prevention,  effectively  eliminating  costly  error  removal  phases  (i.e., 
debugging)  and  produces  verifiably  correct  software  parts.  Development  of  software  proceeds  in  parallel  with  a 
usage  specification  of  the  software.  This  usage  profile  becomes  the  basis  for  a  statistical  test  of  the  software, 
resulting  in  a  scientific  certification  of  the  quality  of  the  software  part  of  the  system. 

A  quick  high  level  comparison  between  the  typical  development  and  CSE  philosophy  of  software 
development  is  summarized  in  Table  I.  The  typical  development  environment  can  be  characterized  by  craft  based 
techniques  which  are  highly  dependent  upon  the  skills  of  the  individuals  involved  whereas  CSE  is  an  engineering 
discipline  with  associated  rigor  and  formality. 

Table  I:  Comparison  between  Typical  Development  and  CSE 


Characteristic 

Typical  Development 

CSE 

Programs  regarded  as 

Lines  of  Instructions 

Correct  rule  for  a  function. 

Specification  focus 

Incomplete  description  of 
external  behavior  and  internal 
design  details. 

Complete,  precise  description  of 
external  behavior;  design  details 
left  for  development. 

Specification  to  code 
transformation  process 

Informal,  debugging  to  verify 
code. 

Stepwise  refinement  and 
verification  using  Box 

Structures. 

Failures  are 

Expected  and  accepted. 

Unacceptable. 

Testing  strategy 

Futile  attempt  for  coverage  and 
little  insight  on  field  reliability. 

Random  sample  based  on  usage 
model  that  predicts  field 
reliability. 

Description  of  Demonstration 


To  conduct  the  demonstration,  both  control  and  demonstration  groups  were  identified.  The  control  group 
consisted  of  a  sample  set  of  ongoing  and  completed  software  projects  at  the  LCSEC.  These  projects  represent  the 
use  of  "typical”  software  engineering  methods  at  the  LCSEC.  Enhancement  projects  at  Picatinny  typically  include 
the  correction  of  observed  problems,  the  addition  of  new  capabilities,  and  in  some  cases,  re-engineering  of 
software.  The  demonstration  project  was  the  Mortar  Ballistics  Computer  (MBC)  re-engineering  effort.  The 
demonstration  aspect  of  this  project  was  the  adoption  of  the  CSE  technology  techniques  as  provided  by  the 
participation  of  Loral  and  SET.  The  hypothesis  to  be  confirmed  or  rejected  in  this  demonstration  was:  The  use  of 
CSE  practices  improves  the  effectiveness  (quality  and  productivity)  of  the  LCSEC  software  support  mission. 

In  order  to  transfer  the  technology  to  be  demonstrated,  as  well  as  the  process  and  culture  for  a  Cleanroom 
environment,  four  different  tools  were  employed: 

(1)  training,  in  a  formal  classroom  setting  which  integrated  lecture  material  and  numerous  hands-on 
workshops  (tailored  for  this  effort), 

(2)  coaching,  both  for  project  planning  and  execution  as  well  as  a  medium  to  promote  ongoing  education, 

(3)  process  handbooks  (evolved  for  this  effort),  which  act  as  a  written  source  of  educational  material  and 
as  a  reference  during  project  execution,  and 

(4)  an  automated  process  support  system  (developed  for  this  effort),  that  helps  enforce  process  adherence 
and  monitors  task  completion,  by  automating  non  creative  tasks. 


Project  Overview 


The  MBC  project  was  a  re-engineering  of  the  current  system  used  by  the  US  Army  to  aim  mortars  for 
combat  support.  The  existing  MBC  was  implemented  in  DTL  (Display  Terminal  Language)  and  Z-80  Assembler 
and  is  not  easily  upgraded  for  new  requirements.  The  re-engineered  system  is  being  implemented  in  Ada  and  as  a 
result  can  be  moved  to  new  updated  hardware  platforms.  Initial  Cleanroom  training  was  provided  in  November  of 
1992  and  January  of  1993,  and  the  MBC  re-engineering  started  in  February  of  1993.  The  first  two  increments  of 
the  MBC  were  completed  by  January  1994. 

In  November  of  1993,  the  LCSEC  participated  in  a  proposal  to  take  on  the  responsibility  for  the 
Improved  MBC  (I-MBC)  system,  which  will  replace  the  MBC  currently  in  the  field.  The  other  partners  in  this 
proposal  were  other  organizations  at  Picatinny  Arsenal.  Picatinny  had  some  key  win  themes  that  were  achieved 
through  the  initial  STARS  cooperation.  They  had  some  of  the  functionality  of  the  proposed  I-MBC  already 
developed  and  certified  to  be  of  high  quality.  They  also  had  shown  that  by  using  Cleanroom,  they  could  develop 
software  of  high  quality  with  extremely  high  productivity.  This  helped  support  the  commitment  that  the  Picatinny 
Arsenal  organization  would  be  successful  with  the  I-MBC  project  and  could  do  so  within  acceptable  budget  limits. 

Picatinny  won  the  bid  in  January  of  1994  and  started  work  on  the  I-MBC  in  February  1994.  The  I-MBC 
will  replace  the  current  M23  Mortar  Ballistics  Computer,  and  will  run  on  a  ruggedized  notebook  computer.  In 
addition,  the  I-MBC  will  add  capabilities  from  other  related  systems  that  could  not  be  accommodated  by  the  M23 
computer.  The  I-MBC  work  began  with  the  development  of  MBC  increment  3  which  did  not  impact  the  I-MBC 
effort  because  the  functions  performed  by  the  MBC  are  a  subset  of  those  to  be  performed  by  the  I-MBC.  Starting 
with  increment  4,  the  LCSEC  team  decided  that  specification,  development  and  certification  for  I-MBC  were  to 
be  created  in  an  incremental  manner,  similar  to  the  three  MBC  increments.  In  order  to  do  this,  some  risk  would 
be  entailed,  since  it  was  necessary  to  make  sure  the  specifications  for  one  increment  did  not  limit  options  for  a 
future  increment.  For  that  reason,  the  I-MBC  team  focused  on  creating  a  ‘system’  specification,  as  well  as 
increment  specifications.  In  this  manner,  risk  was  minimized,  since  increment  behavior  was  checked  to  be 
consistent  with  requirements  defined  in  the  ‘system’  specification. 


The  transfer  of  CSE  technology  was  achieved  through  formal,  classroom-style  training  courses  and 
follow-on  coaching  of  demonstration  team  members.  The  courses  involved  instruction  on  the  underlying 
specification,  development,  and  certification  methods  of  CSE  and  included  in-class  workshops  so  that  students 
gained  experience  applying  the  technology.  As  often  as  possible,  workshops  were  supplemented  with  examples 
extracted  from  the  MBC  project.  Training  provided  the  introduction  to  and  initial  experience  with  the  tools  that 
would  help  enhance  individual  and  team  performance. 

Project  support  was  given  to  the  team  members  through  repeated  on-site  coaching  visits  by  CSE  experts 
from  Loral  and  SET.  This  activity  helped  to  solidify  the  new  ideas  as  team  members  saw  how  the  techniques 
were  applied  to  their  specific  problems. 

The  major  intent  of  the  training  and  coaching  was  to  establish  the  human  behavioral  changes  necessary  to 
develop  better  software.  Implementing  CSE  is  an  intellectually  challenging  process  that  instills  specific  values 
into  its  participants.  For  example,  the  focus  on  product  quality,  a  major  Cleanroom  theme,  instills  a  "get  it  right 
the  first  time"  attitude  into  the  members  of  CSE  teams.  As  successes  were  made  and  milestones  conquered,  the 
CSE  teams  reported  significant  improvements  in  job  satisfaction,  team  spirit,  and  the  desire  to  continue  quality 
improvements.  A  significant  focus  of  the  coaching  effort  was  to  positively  reinforce  each  project  success  in  order 
to  create  a  stronger  identity  with  the  project. 

Such  behavioral  changes  within  a  project  are  improved  by  active  participation  from  all  levels  of  the 
organizational  hierarchy  from  contributing  technical  leads  to  engineering  management.  The  initial  plan  was  for 
the  project  staffs  to  work  closely  as  teams,  rather  than  as  individuals.  Additionally,  the  intention  was  for  the  staffs 
to  be  motivated  and  excited  about  what  they  were  doing;  that  is,  have  a  strong  identity  with  the  process  and 
project.  Thus,  coaching  contained  a  "cheer  leading"  aspect,  designed  to  create  a  healthy  Cleanroom  environment. 

Reinforcement  of  CSE  was  provided  through  the  availability  of  a  six  volume  set  of  process  manuals  to 
the  demonstration  groups.  These  process  manuals  were  an  integral  part  of  the  training  program  and  were 
discussed  in  detail,  both  during  the  formal  training  sessions  and  off-line  as  a  part  of  the  follow-on  coaching 
activities.  The  process  manuals  augment  the  training  by  providing  reference  information  to  LCSEC  engineers, 
using  Cleanroom  concepts,  and  serve  as  a  reference  source  for  resolving  questions  about  specific  issues 
concerning  process  adherence.  The  process  manuals  are  organized  as  follows: 

Volume  1:  Cleanroom  Engineering  Process  Introduction  and  Overview 

Volume  2:  Organization  and  Project  Formation  in  the  Cleanroom  Environment 

Volume  3:  Project  Execution  in  the  Cleanroom  Environment 

Volume  4:  Specification  Team  Practices 

Volume  5:  Development  Team  Practices 

Volume  6:  Certification  Team  Practices 

The  division  of  the  volumes  represents  a  separation  of  concerns  for  the  various  project  stakeholders.  Each  volume 
defines  the  tasks  and  the  control  flow  between  the  tasks  necessary  to  conduct  the  specific  process  which  is  the 
focus  of  the  manual.  Engineering  processes  are  defined  as  formal  control-flow  procedures  with  specific 
completion  conditions.  Collections  of  engineering  processes  also  have  the  same  level  of  formalized  control  flow 
and  completion  conditions.  Thus,  each  engineer,  manager  or  other  staff  member  has  well  defined  roles  and  tasks 
that  exist  as  a  part  of  a  larger  software  process. 

CSE  is  a  formal  process  that  clearly  defines  the  tasks  necessary  for  the  engineering  effort  to  progress,  the 
completion  conditions  for  each  task,  and  the  control  flow  that  dictates  the  order  of  work  on  each  task.  Process 
management  entails  the  use  of  a  clearly  defined  process  as  the  approach  to  be  used  to  complete  the  particular 
project.  The  intent  with  process  management  is  to  give  engineers  a  clear  and  understandable  road  map  which  they 
can  follow  and  by  which  they  may  track  progress  towards  project  completion. 

Awareness  of  software  process  is  a  key  issue  in  successfully  transferring  technology  to  an  organization 
and  to  an  organization’s  long  term  success  with  applying  CSE.  The  project  staffs  at  the  Picatinny  LCSEC 
received  an  introduction  to  process  definition  and  process  management  in  the  context  of  CSE.  Coaching  also 
reinforced  the  importance  of  following  the  defined  process  and  using  the  process  definition,  which  defines  the 
possible  project  alternatives,  to  support  the  selection  of  correct  project  choices. 


Baseline  Metrics 


The  control  groups  represent  the  state-of-the-practice  at  the  LCSEC.  Baseline  metrics  were  collected  in 
order  to  gain  insight  into  project  practices  and  to  establish  a  basis  of  comparison  to  the  demonstration  Cleanroom 
groups.  The  Control  Group  Projects  consisted  of  software  engineering  efforts  in  Fortran,  Z-80  Assembler, 
Motorola  68000  Assembler  and  DTL  (Display  Terminal  Language).  A  LOC  is  defined  as  a  carriage  return  for 
these  baseline  control  group  projects.  This  definition  was  used  because  of  the  wide  difference  in  software 
development  languages  contained  within  the  control  group  projects.  Table  II  presents  the  baseline  metrics  for  the 
control  group.  These  metrics  are  presented  with  the  caution  that  some  data  collection  mechanisms  are  unreliable, 
resulting  in  inaccuracies.  The  numbers  in  Table  II  are  similar  to  results  reported  by  Mosemann  for  other  projects 
within  the  DoD  rAda  and  C++:  A  Business  Case  Analysis.  July  1991]. 

Table  K:  Baseline  Metrics  for  Control  Group  Projects 


Project  /  Measure  . 

Control  Group  Projects 

Number  of  Projects 

5 

Range  of  Effort  -  Staff  Months 

21-58 

Total  Technical  Staff  Months 

192 

Total  KLOC  (*) 

23.14 

DERIVED  METRIC: 

121 

Productivity  -  LOC/Staff  Month 

Observations 

The  following  observations  are  a  compilation  of  experiences  with  the  MBC  and  I-MBC  teams.  These 
observations  are  in  the  context  of  Loral’s  and  SET's  other  experiences  with  replacing  craft-based  practices  with 
engineering-based  practices,  both  in  the  private  sector  and  with  government  organizations.  One  must  keep  in 
mind  these  observations  are  preliminary  since  the  project  has  not  been  completed. 

1.  The  assigned  project  teams  were  able  to  assimilate  and  even  adapt  the  Cleanroom  Software 
Engineering  practices. 

A  common  worry  among  managers  when  hearing  about  Cleanroom  is  that  it  is  too  hard  or  too 
mathematical  for  their  staff.  At  Picatinny,  engineers  were  able  to  apply  and  adapt  the  Cleanroom  practices  to  the 
needs  of  their  project.  Engineers  learned,  used  and  extended  the  ideas  successfully  for  their  project.  The  evidence 
of  this  observation  is  the  products  they  have  produced. 

Disciplined  engineering  in  a  team  environment  requires  rigor,  cooperation  of  individuals,  and  the 
creativity  to  apply  theory  to  real  world  problems.  This  creates  a  challenging  work  environment  that  tends  to  bring 
out  the  best  in  both  individuals  and  teams. 

A  prime  example  of  the  accomplishments  of  the  MBC  team  was  the  tailoring  of  the  box  structures 
algorithm  to  meet  both  their  application  environment  and  the  target  programming  language,  Ada.  MBC  team 
members  have  made  original  contributions  to  the  expression  of  box  structure  constructs  in  Ada,  which  will  have 
applicability  across  many  Cleanroom  projects.  This  has  benefited  both  the  project,  in  terms  of  constructive 
methods,  and  the  individual  team  members,  in  terms  of  a  sense  of  accomplishment.  Building  on  this 
accomplishment,  additional  improvements  for  documenting  the  black  box  were  later  determined.  The 
development  team  felt  that  the  initial  specification  approach  used  for  the  first  three  increments,  was  insufficient, 
in  that  the  information  provided  was  incomplete  and  not  easily  readable.  Through  a  great  deal  of  interaction 


between  the  specification  and  development  teams,  the  specification  team  adopted  a  documentation  approach  that 
fulfilled  the  development  team’s  requirements. 

The  team  has  enjoyed  using  the  various  Cleanroom  techniques  and  have  seen  many  real 
accomplishments.  The  specification  team  is  convinced  that  this  is  the  most  complete  and  precise  specification 
they  have  ever  seen.  The  step-wise  refinement  and  verification,  which  drives  engineers  to  define  one  small  step  to 
take  at  a  time,  take  that  step,  and  then  confirm  its  correctness,  has  also  been  successful.  The  development  team  is 
convinced  that  they  have  a  great  design  and  have  minimized  the  amount  of  code  they  need  to  develop. 

2.  A  process  driven  approach  supports  engineers  in  mastering  a  new  technology. 

Process  driven  project  management  is  one  of  the  two  basic  technologies  being  advanced  by  the  STARS 
program.  The  Picatinny  project  was  the  first  project  on  which  this  key  idea  has  been  employed. 

The  reason  process  driven  management  seems  to  support  technology  transfer  can  be  summarized  as 
follows.  When  doing  something  for  the  first  time,  one  often  asks,  "What  do  I  do  next?"  or  "When  will  I  be  done?" 
This  indicates  a  lack  of  understanding  the  big  picture,  where  engineers  can  clearly  place  their  efforts  in  a  project 
context.  This  is  not  only  an  attribute  of  first  time  usage  of  techniques  or  a  process,  but  also  an  indication  that  a 
clearly  defined  process  does  not  exist  or  is  not  effectively  managed. 

By  placing  the  Cleanroom  techniques  within  a  fully  defined  process,  LCSEC  engineers  knew  precisely 
what  step  they  were  currently  on,  as  well  as  what  had  been  completed  and  what  remained  to  be  done.  Giving  each 
individual  the  foresight  that  showed  where  they  were  in  the  context  of  the  entire  project  strengthened  project 
identity  and  boosted  morale. 

3.  Staff  morale  has  improved  on  the  project  teams. 

Another  common  fear  of  managers  when  hearing  about  Cleanroom  is  that  their  staff  members  will  not 
like  it  due  to  the  rigor  of  the  process  and  the  absence  of  “positive”  feedback  through  executing  their  own  code. 
This  has  not  the  been  the  case  at  other  places  where  cleanroom  has  been  introduced,  and  Picatinny  is  no 
exception.  When  an  organization  replaces  craft-based  practices  with  engineering-based  practices,  morale 
improves.  One  reason  seems  to  be  that  people  now  know  what  to  do,  when  to  do  it,  and  how  it  should  be  done. 
This  eliminates  the  uncertainty  and  anxiety  that  results  when  one  has  to  not  only  do  a  job  but  also  has  to  determine 
what  to  do  and  when.  Additionally,  when  engineers  learn  to  use  the  Cleanroom  practices,  they  know  they  can  do 
the  high  quality  job  they  have  been  striving  to  achieve.  Engineers  are  convinced  that  they  are  producing  a  better 
product.  As  a  result,  they  are  excited  about  it. 

At  the  Picatinny  LCSEC,  all  the  engineers,  both  in  informal  contacts  and  in  a  questionnaire  distributed  to 
the  engineering  staff,  reported  morale  improvements.  The  LCSEC  management  has  also  confirmed  the  existence 
of  the  improved  morale  and,  of  course,  is  favorably  impressed. 

4.  Facilitation  of  work  effort  is  greatly  enhanced  through  process  driven  management. 

Team  leaders  managed  by  process  definition  and  task  lists  which  allowed  more  visibility  of  project  status 
by  management.  The  intent  with  process  management  is  to  give  engineers  a  clear  and  understandable  road  map 
which  they  can  follow  and  by  which  they  may  track  progress  towards  project  completion.  Awareness  of  software 
process  is  a  key  issue  in  successfully  transferring  technology  to  an  organization  and  to  an  organization's  long  term 
success  with  applying  a  given  process  driven  approach  to  project  management. 

5.  The  team-oriented  approach  of  CSE  saw  immediate  acceptance  and  realized  both  tangible  and 
intangible  benefits. 

A  key  ingredient  of  Cleanroom  is  that  a  team  amplifies  human  performance.  People  took  advantage  of 
the  insight  of  others  in  order  to  bring  about  the  best  possible  project  result.  Good  people  working  together 
produce  better  results.  The  simple  idea  that  many  minds  are  better  than  one  makes  the  outlook  for  quality  good. 
However,  some  less  tangible  benefits  were  realized  as  well.  The  fact  that  the  entire  team  is  responsible  for 
quality,  in  a  series  of  checks  and  reviews,  puts  pressure  on  the  team  and  not  on  individuals.  This  pressure  creates 


a  reliance  on  team  activity  over  individual  performance.  Furthermore,  as  successes  are  encountered,  the  entire 
team  takes  credit,  not  a  single  individual,  thus,  cementing  the  teamwork  concepts.  The  bottom  line  is  that 
teamwork  improves  individual  performance. 

Our  observation  is  that  the  I-MBC  team  now  works  within  an  effective  team-oriented  environment.  We 
believe  that  further  use  of  Cleanroom  will  establish  a  strong  team  mentality  that  will  serve  to  further  improve  the 
initial  good  results. 

6.  Coaching  is  a  key  ingredient  of  technology  transfer  success. 

Although  the  training  was  rigorous  with  a  mixture  of  theory  and  workshops,  students  learn  at  different 
rates.  Coaching  allowed  Loral  and  SET  staff  to  re-educate  the  slower-to-adopt  project  staff  members  and  keep  the 
entire  team  on  a  common  level  of  knowledge  and  expertise.  Loral  and  SET  technical  presence  at  project  inception 
and  during  project  execution  helped  solidify  the  transfer  of  the  technology  and  ensured  that  the  project  got  started 
in  the  most  efficient  manner. 

Furthermore,  there  was  a  gap  between  the  end  of  training  and  the  start  of  the  project  and  some  of  the 
education  was  forgotten.  Coaching  became  the  mechanism  to  re-educate  and  supplement  the  original  training. 
Further,  as  good  ideas  were  conceived  by  some  team  members,  it  was  possible  to  see  that  all  members  were 
supplied  with  the  new  ideas. 

As  the  project  progressed,  the  CSE  ideas  needed  to  be  adapted  to  the  specific  Picatinny  environment. 
Coaches  were  used  to  discuss  design  alternatives  and  to  help  in  refining  the  technology  to  best  serve  the 
application. 

Perhaps  the  most  unnoticed  but  effective  use  of  coaching  was  in  the  positive  reinforcement  the  CSE 
coaches  were  able  to  give  to  the  team  members  and  the  team  as  a  whole.  Coaches  are  recognized  as  experts. 
When  experts  comment  positively  on  original  ideas  by  a  team  member,  the  effect  can  be  enormous  in  terms  of 
self-esteem  and  sense  of  accomplishment  and  contribution.  The  CSE  trainers  tried  to  positively  reinforce  the 
behavior  of  those  making  such  contributions  and  encourage  others  to  seek  answers  beyond  the  limits  of  current 
knowledge.  The  "cheer  leading"  approach  increased  project  satisfaction,  which  motivated  greater  project 
performance.  Based  on  this  experience,  it  is  now  believed  that  coaching  should  be  a  formal  part  of  any 
technology  transfer  effort. 

7.  Communication  among  teams  (and  between  team  members )  is  greatly  enhanced  through  a  process 
driven  CSE  approach. 

An  important  ingredient  of  any  process  driven  activity  is  communication  among  contributing  teams  and 
individuals.  One  aspect  of  this  was  that  no  team  culture  existed  at  the  LCSEC;  meaning  that  no  real  notion 
existed  of  how  teams  are  supposed  to  behave  during  project  execution.  This  problem  manifested  itself  in  many 
different  ways.  Testers  often  did  not  receive  specification  updates  (and  failed  to  ask  for  them).  Also,  work  tended 
to  be  duplicated  by  multiple  personnel  because  the  division  of  tasks  was  unclear  and  communication  among 
members  occurred  too  seldom. 

Two  approaches  were  used  to  solve  this  problem  at  Picatinny.  The  first  was  to  establish  effective 
communication  among  team  members  and  the  second  was  to  establish  communication  among  the  different 
departments  involved  in  the  project.  The  adoption  of  a  well  defined  process  includes  a  vocabulary  that  is  of  great 
help  to  the  understanding  and  discussion  of  the  process.  This  well  defined  vocabulary  makes  communication 
between  team  members  much  more  effective  and  productive.  The  improved  communication  also  started  a  shift  in 
the  culture  of  the  teams.  Team  members  report  that  they  readily  use  each  other  as  information  sources,  quality 
checks,  etc.  Team  reviews  are  effective  and  informative.  However,  the  second  aspect,  communication  between 
departments,  continues  to  be  a  problem.  The  MBC  certification  team  members  work  for  a  different  department 
than  the  specification  and  development  teams.  Resulting  problems  are  that  the  certification  team  finds  themselves 
working  from  outdated  specifications.  Furthermore,  the  certification  team  seems  to  duplicate  each  other's  work. 
A  future  goal  is  to  be  able  to  duplicate  the  success  of  the  specification  and  development  teams  in  the  certification 
team,  primarily  by  improving  communications.  A  more  concerted  effort  should  have  been  made  by  the  coaches  to 
minimize  these  communications  problems. 


8.  Specific  techniques  ofCleanroom  Software  Engineering  were  easily  and  successfully  used. 

Three  specific  techniques  were  identified  by  project  staff  as  being  major  sources  of  their  improved 
performance.  These  techniques  are  team  reviews,  Cleanroom  specifications  and  box  structured  design,  and  are 
described  in  greater  detail  below: 

Team  reviews,  although  slow  and  awkward  at  the  start,  were  cited  by  team  members  as  one  of 
the  most  successful  aspects  of  CSE.  Members  report  that  “team  shared  responsibility”  eased 
misgivings  about  participating  in  such  a  big  project.  This  negated  "finger  pointing"  that  existed 
in  previous  projects  and  allowed  even  difficult  personality  combinations  to  work  together.  The 
result  was  that  everyone  participated  and  worked  as  a  team  toward  project  success  and 
completion.  Morale  increased  sharply  as  groups  of  individuals  transformed  into  an  effective 
software  team. 

Cleanroom  specification,  most  notably  black  box  documentation,  was  cited  as  being 
responsible  for  gains  in  productivity.  Many  talented  engineers  existed  on  the  project  and  their 
productivity  was  significantly  enhanced  when  working  from  a  well  defined  problem  statement. 

The  completeness  of  the  specification  was  the  main  reason  cited  for  the  team's  confidence  that 
they  were  producing  a  high  quality  product. 

Box  structured  design  is  credited  with  focusing  the  code  generation  process  and  with  making 
team  reviews  more  effective.  The  team  enjoyed  the  orderly  process  of  developing  software.  It 
got  them  started  more  quickly  on  solving  a  particular  problem  and  they  were  able  to  measure  the 
progress  of  the  development  activity  with  more  precision  than  in  the  past.  Since  the  process 
relies  a  great  deal  on  logical  thinking  as  opposed  to  programming  skill,  less  experienced 
programmers  are  able  to  take  a  bigger  share  of  the  development  burden.  Therefore,  software 
engineers  can  make  the  most  of  their  software  engineering  skills  without  having  to  develop  in- 
depth  programming  language  expertise. 


Results 

The  most  important  result  noted  by  this  effort,  even  in  its  preliminary  form,  is  that  the  motivation  to 
continue  to  use  Cleanroom  practices  at  Picatinny  has  been  established.  This  demonstration  effort  was  sponsored 
by  STARS  and  the  continued  effort  is  being  sponsored  by  the  TACOM  LCSEC.  This  result  is  an  instance  of  the 
STARS  program  fulfilling  its  mission  by  being  the  catalyst  for  introducing  improvements  to  the  software 
engineering  capabilities  in  the  DoD.  The  decision  to  expand  CSE  usage  across  the  entire  organization,  is  the  most 
definitive  conclusion  of  this  effort. 

In  addition  to  the  above  mentioned  conclusion,  the  following  conclusions  can  be  drawn  based  on  the 
current  status  of  the  MBC/l-MBC  project. 

1.  It  is  possible  to  transfer  CSE  practices  to  project  teams  operating  within  a  typical  immature  DoD  SSA 
organization . 

This  was  shown  by  the  fact  that  the  MBC  and  I-MBC  project  has  progressed  to  a  point  where  CSE  is 
being  successfully  applied.  This  result  shows  that  a  specific  CMM  maturity  rating  is  not  necessary  in  order  to 
benefit  from  Cleanroom  Software  Engineering.  The  engineering  staff  also  enjoyed  using  the  ideas,  and  all  were 
interested  in  using  the  ideas  again.  Additionally,  nearly  all  were  interested  in  supporting  and  participating  in  the 
establishment  of  the  planned  "Cleanroom  Competency  Center"  at  the  Picatinny  Arsenal. 

2.  Typical  DoD  SSA  organizations  can  realize  important  benefits ,  in  terms  of  improved  process 
productivity ,  product  quality ,  and  staff  morale,  from  the  application  of  CSE. 


This  conclusion  is  supported  by  the  tripling  of  productivity  of  the  MBC  team.  The  LOC  were  calculated 
the  same  way  for  this  Ada  Language  based  code  as  was  done  for  the  baseline  control  group  projects  with  the 
exception  that  blank  lines  of  code  were  not  counted.  Table  III  shows  the  results  for  productivity  for  increments  1 
through  4.  Table  IV  shows  the  raw  numbers  that  were  used  to  calculate  the  productivity  shown  in  Table  III. 
Future  increments  should  show  additional  improvement. 

Table  III:  Productivity  Change  for  MBC/I-MBC 


Increment 

LOC  per  Staff 
Month 

Change  in  Productivity  based 
on  Picatinny  Baseline  Metrics 

1 

370 

3.1 :  1 

!  1  +  2 

519 

4.3 : 1 

1  +  2  +  3 

see  note 

see  note 

1+2+3+4 

443 

3.7:  1 

Note:  Staff  months  data  lost  due  to 


accounting  mistake  but  included  in  next  entry 


Change  in  Productivity  =  LOC  per  Staff  Month _ 

121  LOC  Baseline  Productivity 


Table  IV:  Cumulative  Raw  Numbers  for  MBC/I-MBC 


Increment 

MBC 

Staff  Months 

Coaching  Staff 
Months 

Training  Staff 
Months 

Lines  of  Code 
(LOC) 

1 

23 

3 

5 

8,500 

1+2 

42.5 

6 

5 

22,063 

1-3 

see  note 

8.5 

5.5 

28,579 

Table  III 

1-4 

86.1 

10 

6 

38,155 

Early  indications  are  that  quality  appears  to  have  improved  over  previous  product  quality  according  to 
Picatinny's  customers.  The  quality  of  the  software  (See  Table  V)  developed  is  very  high  when  compared  to 
quality  for  traditional  software  development.  Thus,  the  result  achieved  will  be  viewed  by  the  team  as  the  mark  to 
better  on  the  next  increment  of  this  project.  The  incentive  and  motivation  for  continual  improvement  is  firmly  in 
place  among  the  team  members. 


Table  V:  Software  Quality 


Failure 

Type 

Increment  Failures 

Description 

D 

2 

3 

4 

Process 

19 

6 

0 

0 

Approved  improvements  made  to  design 
but  not  reflected  in  updates  to  the 
specifications  for  certification. 

Spelling 

3 

0 

1 

0 

Misspellings  on  displays. 

Behavior 

3 

3 

6 

Coding  Errors,  did  not  work  as  specified. 

Total 

24 

9 

■ 

6 

Total  numbers  of  failures 

Quality 

2.82 

.41 

.14 

.16 

Failures/KLOC 

The  MBC  project  has  realized  significant  gains  from  the  CSE  ideas.  Once  the  learning  curve  had  been 
completed,  initial  successes  in  creating  the  Black  Box  specification  served  to  cement  commitment  to  CSE. 

The  resulting  conclusions  from  the  overall  evaluation  are  preliminary  because  the  demonstration  project 
is  still  in  its  early  stages.  However,  the  original  hypothesis  that  Cleanroom  improves  the  effectiveness  of  the 
software  re-engineering  activities  at  Picatinny  looks  very  promising.  Indeed,  management  and  staff  agree  that 
morale  and  motivation  is  extremely  high,  that  teamwork  is  now  the  normal  mode  of  operation,  and  that  people  are 
excited  about  the  software  process  being  established  and  are  motivated  to  produce  high  quality  products. 

A  good  technical  road  map  is  in  place  at  Picatinny;  the  technical  personnel  are  developing  the  skills  that 
appear  to  show  significant  gains  in  productivity.  Even  more  promising  is  the  fact  that  these  gains  were  made  on 
the  first  use  of  CSE 

3.  The  return  on  investment  at  Picatinny  cannot  be  definitively  calculated ,  but  indications  are  that  there 
is  a  significant  return  on  investment . 

Since  the  project  is  not  yet  complete,  a  preliminary  estimate  of  return  on  investment  can  only  be  based  on 
estimates  from  the  information  currently  available.  The  resulting  return  on  investment  (ROI)  calculations  appear 
below  in  Table  VI. 


Table  VI:  Return  on  Investment  for  MBC/I-MBC 


Increment 

LOC  per 
Staff 
Month 

Staff  Months  MBC  would  have 
taken  without  CSE 

ROI 

Base 

ROI 

with 

Coaching 

1 

370 

8500  LOC/121  LOC/SM  =  70.3 

5.9:1 

8.9:1 

1  +  2 

519 

22063  LOC/121  LOC/SM  = 
182.3 

12.7:1 

26.7:1 

1-4 

443 

38155  LOC/121  LOC/SM  = 

315.3 

14.3:1 

36.5:1 

SM  =  Staff  Months 


ROI  Base  =  SM  MBC  would  have  taken  -  SM  MBC  took 

Staff  Months  of  Coaching  Effort  +  Staff  Months  spent  in  Training 


ROI  with  Coaching  =  SM  MBC  would  have  taken  -  (SM  MBC  took  +  SM  of  Coaching  Effort) 

Staff  Months  spent  in  Training 


4.  An  Automated  Process  Support  System  (PSS),  that  is  consistent  with  the  process  defined  for  the 
project,  facilitates  technology  transfer 

Automating  the  non-creative  tasks  of  a  new  technology,  such  as  file  access  and  simple  process  flow 
facilitates  the  adoption  of  the  new  technology. 

5.  Based  on  this  demonstration  we  now  believe  that  a  technology  transfer  program  to  support  individual 
projects  at  a  typical  DoD  SSA  organization  must  begin  with  a  defined  process  for  the  project  and  should  consist 
of  the  following  components: 

(1)  a  technology  transfer  plan, 

(2)  formal  CSE  training  , 

(3)  the  availability  of  qualified  coaching, 

(4)  the  availability  of  engineering  handbooks. 

The  combination  of  technology  transfer  components  created  a  series  of  successes  at  Picatinny;  including 
productivity  gains,  expected  quality  gains,  and  the  increased  motivation  of  the  engineering  staff. 


Future  Directions 

The  next  steps  for  LCSEC/STARS  cooperation  have  been  defined.  The  impressive  results  to  date  in  the 
areas  of  productivity,  quality,  return  on  investment  and  moral  have  convinced  TACOM  LSCEC  management  to 
continue  the  work  begun  under  this  demonstration  project  and  to  expand  it  further  throughout  the  organization  and 
this  includes  the  following: 

Creating  a  Cleanroom  Competency  Team  to  build  CSE  and  process  expertise  in  house.  This  group  will 
be  responsible  for  continuous  review  and  study  of  the  application  of  CSE  at  Picatinny. 


Evolve  Cleanroom  Software  Engineering  into  a  complete  life  cycle  process  as  far  as  is  feasible  using 
standard  Cleanroom  techniques.  Work  is  currently  underway  to  perform  a  mapping  of  the  evolved  Cleanroom 
Software  Engineering  process,  as  used  at  Picatinny,  against  SEI  developed  Software  Process  Frameworks  (SPF) 
for  the  SEI  CMM  Levels  2  and  3.  The  results  of  this  mapping  will  provide  Picatinny  with  a  road  map  of  all  areas 
that  are  either  not  covered  or  are  poorly  covered  by  the  currently  documented  CSE  process.  These  results  will  be 
used  to  identify  areas  for  Cleanroom  process  definition  extensions.  Areas  that  are  not  practical  for  Cleanroom 
extension  will  be  covered  with  non  Cleanroom  processes.  The  result  will  be  a  complete  life  cycle  process 
definition  for  Picatinny  to  support  their  move  to  SEI  CMM  level  3. 


Why  Isn’t  Cleanroom  the  Universal  Software 
Development  Methodology? 

Johnnie  Henderson 
Loral  Space  Information  Systems 


Why  Isn't  Cleanroom  the  Universal  Software  Development  Methodology? 

Johnnie  Henderson,  Loral  Space  Information  Systems 

Cleanroom  -  a  methodology  that  promises  much  lower  error  rates,  higher  productivity,  delivery  of 
software  on  schedule  and  within  budget  -  sounds  like  the  proverbial  "silver  bullet"  that  the  industry  is 
looking  for.  Why  hasn't  it  caught  on  and  spread  like  wildfire?  There  are  three  basic  reasons:  a  belief 
that  the  Cleanroom  methodology  is  too  theoretical,  too  mathematical  and  too  radical  for  use  in  real 
software  development,  it  advocates  no  unit  testing  by  developers  but  instead  replaces  it  with 
correctness  verification  and  statistical  quality  control--  concepts  that  are  directly  opposite  of  how  most 
software  is  developed  today,  and  maturity  of  the  software  development  industry.  Use  of  Cleanroom 
processes  requires  rigorous  application  of  defined  processes  in  all  lifecycle  phases.  Since  most  of  the 
industry  is  still  operating  at  the  ad  hoc  level  (as  defined  by  the  SEI  CMM),  the  industry  has  not  been 
ready  to  apply  those  techniques. 

What  does  the  experience  in  using  Cleanroom  say  about  whether  or  not  these  are  valid  concerns? 
Following  is  a  brief  discussion  of  those  concerns: 

Correctness  verification  is  too  theoretical  to  be  usable  in  real  software  development.  The 

fundamental  approach  to  verification  as  espoused  by  Cleanroom  is  aimed  at  introducing 
mathematical  reasoning,  not  mathematical  notation  into  the  verification  process.  The  principal 
motivation  is  to  provide  a  rigorous  methodology  for  software  development  and  to  provide  a 
firm  foundation  as  an  engineering  discipline.  Mathematical  verification  of  programs  is  done  by 
using  a  few  basic  control  structures  and  defining  proofs  following  rules  specified  in  a 
correctness  theorem.  The  proof  strategy  is  divided  into  small  parts  that  easily  accumulate  into 
proof  for  a  large  software  system!  1], 

The  method  of  human  mathematical  verification  used  in  Cleanroom  is  called  functional 
verification.  Functional  verification  is  organized  around  correctness  proofs,  which  are  defined 
for  the  design  constructs  used  in  a  software  design.  Using  this  type  of  functional  verification, 
the  verification  problem  changes  from  one  with  an  infinite  number  of  combinations  to  consider 
to  a  finite  process  because  the  correctness  theorem  defines  the  required  number  of  conditions 
that  must  be  verified  for  each  design  construct  used.  It  reduces  software  verification  to 
ordinary  mathematical  reasoning  about  sets  and  functions[2].  The  objective  is  to  develop 
designs  in  concert  with  associated  correctness  proofs.  Designs  are  created  with  the  objective  of 
being  easy  to  verify.  A  rule  of  thumb  followed  is  that  when  designs  become  difficult  to  verify, 
they  should  be  redone  for  simplicity!  1,2]. 

The  Cleanroom  methodology  has  been  used  to  develop  a  variety  of  types  of  applications,  most 
of  the  applications  in  software  that  has  been  sold  commercially  or  embedded  in  operational 
application  systems[3].  Individuals  involved  in  these  development  efforts  were  a  cross  section 
of  the  software  engineering  population  and  were  able  to  apply  the  Cleanroom  method  after  a 
brief  but  intensive  training  program[4]. 


Unit  testing  vs.  statistical  quality  control.  Statistical  quality  control  is  used  when  you  have 
too  many  items  to  test  all  of  them  exhaustively.  Instead,  you  statistically  sample  and  analyze 
some  items  and  scientifically  assess  the  quality  of  all  of  the  items  through  extrapolation.  This 
technique  is  widely  used  in  manufacturing  in  which  items  in  a  production  line  are  sampled,  the 
quality  is  measured,  then  sample  quality  is  extrapolated  to  the  entire  production  line,  and  flaws 
are  corrected  if  the  quality  is  not  as  expected. 

For  software,  this  notion  has  been  evolved  so  that  you  perform  statistical  usage  testing-testing 
the  software  the  way  the  users  intend  to  use  it.  This  is  accomplished  by  defining  usage 
probability  distributions  that  identify  usage  patterns  and  scenarios  with  their  probability  of 
occurrence.  Tests  are  derived  that  are  generated  based  on  the  usage  probability  distributions. 
System  reliability  is  predicted  based  on  analysis  of  the  test  results  using  a  formal  reliability 
model,  such  as  mean-time-to-failure[4]. 

The  underlying  concern  is  that  random,  statistical-based  testing  will  not  provide  sufficient 
coverage  to  ensure  a  reliable  product  is  delivered  to  the  customer.  The  coverage  concern  stems 
from  a  misapprehension  that  statistical  implies  haphazard,  large,  and  costly  and  that  critical 
software  requirements,  which  may  be  statistically  insignificant,  are  overlooked  or  untested. 
Coverage  is  directly  related  to  the  robustness  of  the  usage  probability  distributions  that  control 
the  selection  process  and  has  not  proven  to  be  a  problem  in  current  applications  of  the 
methods.  In  a  study  performed  by  Dyer  on  the  level  of  requirements  coverage  using  statistical 
testing,  100  per  cent  of  the  high-level  requirements  were  covered,  90  per  cent  of  the 
subcomponent-level  requirements  were  covered,  and  approximately  80  percent  of  all 
requirements  were  covered[4]. 

The  Cleanroom  method  asserts  that  statistical  usage  testing  is  many  times  more  efficient  than 
traditional  coverage  testing  in  improving  the  reliability  of  software.  Statistical  testing,  which 
tends  to  find  errors  in  the  same  order  as  their  seriousness  (from  a  user’s  point  of  view),  will 
uncover  failures  several  times  more  effectively  than  by  randomly  finding  errors  without  regard 
to  their  seriousness.  The  basis  for  software  reliability  starts  with  the  definition  of  a  statsitical 
model,  generally  based  on  the  concept  that  input  data  comes  in  at  random  times  and  with 
random  contents.  With  defined  initial  conditions,  any  such  fixed  use  is  distinguishable  from  any 
other  use.  These  uses  can  be  assembled  into  a  sequence  of  uses,  and  the  collection  identified  as 
a  stochastic  process  subject  to  evaluation  using  statistical  methods. 

Coverage  testing  is  anecdotal  and  can  only  provide  confidence  about  the  specific  paths  tested. 
No  assessment  can  be  made  about  the  paths  not  tested.  Because  usage  testing  exercises  the 
software  the  way  the  users  intend  to  use  it,  high-frequency  errors  tend  to  be  found  early.  For 
this  reason,  statistical  usage  testing  is  more  effective  at  improving  software  reliability  than  is 
coverage  testing.  Coverage  testing  is  as  likely  to  find  a  rare  execution  failure  as  it  is  to  find  a 
frequent  one.  If  the  goal  of  a  testing  program  is  to  maximize  the  expected  mean-time-to-failure, 
hence  the  reliability  of  the  system,  a  strategy  that  concentrates  on  failures  that  occur  more 
frequently  is  more  effective  than  one  that  has  an  equal  probability  of  finding  high-  and  low- 
frequency  failures[5]. 


Human  functional  verification  has  proven  to  be  surprisingly  synergistic  with  statistical  testing 
according  to  Mills,  Dyer,  and  Linger[l].  Experimental  data  from  projects  where  both 
Cleanroom  verification  and  more  traditional  debugging  techniques  were  used  show  that  the 
Cleanroom  verified  software  exhibited  fewer  errors  injected.  Those  errors  were  less  severe  and 
required  less  time  to  fix. 


Software  Process  Maturity  Factor.  Using  the  Cleanroom  methodology  requires  a  change  in 
paradigm-from  viewing  software  development  as  an  art  or  craft  to  viewing  it  as  an  engineering 
discipline.  As  such,  it  must  have  a  rigorous  foundation.  In  other  engineering  disciplines,  failures 
are  neither  expected  nor  accepted  as  normal.  Other  engineering  professions  have  minimized 
error  by  developing  a  sound  theoretical  base  on  which  to  build  design  practices.  Cleanroom 
methods  provide  a  theoretical  foundation  for  a  comprehensive  engineering  process  that  has 
been  reduced  to  practice  for  commercial  software  development. 

Using  Cleanroom  methods  requires  commitments  from  management  to  provide  training  (for 
both  management  and  technical  personnel)  in  the  skills  needed  to  implement  the  methodology. 
It  also  requires  discipline.  Management  must  allow  the  process  to  unfold  naturally,  technical 
personnel  must  rigorously  follow  the  process.  It  may  require  additional  tools,  such  as  some 
automated  support  to  develop  the  randomly  generated  test  suites  from  the  usage  probability 
distributions. 

In  spite  of  these  requirements,  many  have  found  that  the  return  made  the  investments  worthwhile.  If 
being  able  to  develop  high-  quality  software  within  budget  and  on  schedule  are  a  concern,  the 
Cleanroom  methodology  may  be  worth  your  taking  the  time  to  investigate.  Peter  Senge,  in  his  book 
The  Fifth  Discipline^] ,  said  that  30  years  is  the  typical  incubation  period  for  a  basic  innovation,  a 
concept  that  transforms  an  existing  industry.  Maybe  we  are  now  at  the  stage  when  the  Clearoom 
methodology  will  begin  to  take  hold  and  make  that  transformation  of  the  software  development 
industry. 
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